Oct 30 21:32:41.517697 ip-10-0-132-92 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:32:41.908775 ip-10-0-132-92 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:32:41.908775 ip-10-0-132-92 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:32:41.908775 ip-10-0-132-92 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:32:41.908775 ip-10-0-132-92 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:32:41.912716 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912634    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:32:41.912716 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912717    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:32:41.912786 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912721    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:32:41.912786 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912724    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:32:41.912786 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912727    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:32:41.912786 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912731    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:32:41.912786 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912736    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:32:41.912786 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912739    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:32:41.912786 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912742    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:32:41.912786 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912745    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:32:41.912786 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912747    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:32:41.912786 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912750    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:32:41.912786 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912752    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:32:41.912786 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912755    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:32:41.912786 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912757    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:32:41.912786 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912760    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:32:41.912786 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912762    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:32:41.912786 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912765    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:32:41.912786 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912767    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:32:41.912786 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912770    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:32:41.912786 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912776    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:32:41.913237 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912779    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:32:41.913237 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912782    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:32:41.913237 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912784    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:32:41.913237 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912787    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:32:41.913237 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912790    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:32:41.913237 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912793    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:32:41.913237 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912796    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:32:41.913237 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912799    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:32:41.913237 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912801    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:32:41.913237 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912804    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:32:41.913237 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912807    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:32:41.913237 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912809    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:32:41.913237 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912812    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:32:41.913237 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912814    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:32:41.913237 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912817    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:32:41.913237 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912819    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:32:41.913237 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912822    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:32:41.913237 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912825    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:32:41.913237 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912828    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:32:41.913237 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912830    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:32:41.913716 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912833    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:32:41.913716 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912835    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:32:41.913716 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912837    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:32:41.913716 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912840    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:32:41.913716 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912842    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:32:41.913716 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912846    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:32:41.913716 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912849    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:32:41.913716 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912852    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:32:41.913716 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912854    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:32:41.913716 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912857    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:32:41.913716 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912859    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:32:41.913716 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912862    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:32:41.913716 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912865    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:32:41.913716 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912867    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:32:41.913716 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912870    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:32:41.913716 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912873    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:32:41.913716 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912875    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:32:41.913716 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912878    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:32:41.913716 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912881    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:32:41.914151 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912883    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:32:41.914151 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912886    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:32:41.914151 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912889    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:32:41.914151 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912891    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:32:41.914151 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912893    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:32:41.914151 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912896    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:32:41.914151 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912898    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:32:41.914151 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912901    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:32:41.914151 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912903    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:32:41.914151 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912906    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:32:41.914151 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912908    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:32:41.914151 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912912    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:32:41.914151 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912914    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:32:41.914151 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912917    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:32:41.914151 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912920    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:32:41.914151 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912922    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:32:41.914151 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912925    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:32:41.914151 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912927    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:32:41.914151 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912931    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:32:41.914151 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912933    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:32:41.914668 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912936    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:32:41.914668 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912939    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:32:41.914668 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912941    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:32:41.914668 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912944    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:32:41.914668 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912946    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:32:41.914668 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912951    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:32:41.914668 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912954    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:32:41.914668 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912959    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:32:41.914668 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912963    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:32:41.914668 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912966    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:32:41.914668 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912970    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:32:41.914668 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912972    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:32:41.914668 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912975    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:32:41.914668 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912978    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:32:41.914668 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912981    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:32:41.914668 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912984    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:32:41.914668 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912986    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:32:41.914668 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912989    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:32:41.914668 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912993    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:32:41.915106 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912996    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:32:41.915106 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.912998    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:32:41.915106 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913000    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:32:41.915106 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913003    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:32:41.915106 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913006    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:32:41.915106 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913008    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:32:41.915106 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913011    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:32:41.915106 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913825    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:32:41.915106 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913832    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:32:41.915106 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913835    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:32:41.915106 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913838    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:32:41.915106 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913841    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:32:41.915106 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913844    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:32:41.915106 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913847    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:32:41.915106 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913851    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:32:41.915106 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913853    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:32:41.915106 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913856    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:32:41.915106 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913859    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:32:41.915106 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913862    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:32:41.915611 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913864    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:32:41.915611 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913867    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:32:41.915611 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913870    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:32:41.915611 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913873    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:32:41.915611 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913875    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:32:41.915611 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913878    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:32:41.915611 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913881    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:32:41.915611 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913883    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:32:41.915611 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913886    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:32:41.915611 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913889    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:32:41.915611 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913891    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:32:41.915611 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913894    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:32:41.915611 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913897    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:32:41.915611 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913899    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:32:41.915611 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913902    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:32:41.915611 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913904    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:32:41.915611 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913907    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:32:41.915611 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913909    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:32:41.915611 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913912    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:32:41.915611 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913915    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:32:41.916089 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913917    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:32:41.916089 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913920    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:32:41.916089 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913923    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:32:41.916089 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913927    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:32:41.916089 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913931    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:32:41.916089 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913934    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:32:41.916089 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913937    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:32:41.916089 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913939    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:32:41.916089 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913942    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:32:41.916089 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913945    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:32:41.916089 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913948    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:32:41.916089 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913951    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:32:41.916089 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913953    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:32:41.916089 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913956    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:32:41.916089 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913958    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:32:41.916089 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913961    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:32:41.916089 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913964    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:32:41.916089 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913966    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:32:41.916089 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913969    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:32:41.916575 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913973    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:32:41.916575 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913977    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:32:41.916575 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913980    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:32:41.916575 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913983    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:32:41.916575 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913986    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:32:41.916575 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913988    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:32:41.916575 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913991    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:32:41.916575 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913993    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:32:41.916575 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913996    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:32:41.916575 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.913999    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:32:41.916575 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914002    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:32:41.916575 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914004    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:32:41.916575 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914007    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:32:41.916575 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914009    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:32:41.916575 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914012    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:32:41.916575 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914014    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:32:41.916575 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914017    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:32:41.916575 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914019    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:32:41.916575 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914022    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:32:41.916575 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914024    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:32:41.917040 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914027    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:32:41.917040 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914030    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:32:41.917040 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914034    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:32:41.917040 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914036    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:32:41.917040 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914039    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:32:41.917040 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914042    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:32:41.917040 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914044    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:32:41.917040 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914047    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:32:41.917040 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914049    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:32:41.917040 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914052    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:32:41.917040 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914055    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:32:41.917040 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914057    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:32:41.917040 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914062    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:32:41.917040 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914065    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:32:41.917040 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914068    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:32:41.917040 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914070    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:32:41.917040 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914072    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:32:41.917040 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914075    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:32:41.917040 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914077    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:32:41.917040 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914080    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:32:41.918054 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914083    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:32:41.918054 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914085    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:32:41.918054 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914088    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:32:41.918054 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914090    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:32:41.918054 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914093    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:32:41.918054 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914095    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:32:41.918054 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914098    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:32:41.918054 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914100    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:32:41.918054 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914103    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:32:41.918054 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914106    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:32:41.918054 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914108    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:32:41.918054 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914112    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:32:41.918054 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914115    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:32:41.918054 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914118    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:32:41.918054 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914120    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:32:41.918054 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914645    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:32:41.918054 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914652    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:32:41.918054 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914655    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:32:41.918054 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914658    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:32:41.918524 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914660    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:32:41.918524 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914663    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:32:41.918524 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914666    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:32:41.918524 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914668    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:32:41.918524 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914671    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:32:41.918524 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914676    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:32:41.918524 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914679    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:32:41.918524 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914682    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:32:41.918524 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914684    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:32:41.918524 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914687    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:32:41.918524 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914690    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:32:41.918524 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914693    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:32:41.918524 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914695    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:32:41.918524 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914698    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:32:41.918524 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914701    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:32:41.918524 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914704    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:32:41.918524 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914706    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:32:41.918524 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914709    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:32:41.918524 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914711    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:32:41.918954 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914714    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:32:41.918954 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914717    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:32:41.918954 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914719    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:32:41.918954 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914722    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:32:41.918954 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914724    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:32:41.918954 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914728    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:32:41.918954 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914731    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:32:41.918954 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914734    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:32:41.918954 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914736    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:32:41.918954 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914740    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:32:41.918954 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914743    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:32:41.918954 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914745    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:32:41.918954 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914748    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:32:41.918954 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914750    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:32:41.918954 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914753    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:32:41.918954 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914755    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:32:41.918954 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914758    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:32:41.918954 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914761    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:32:41.918954 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914765    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:32:41.918954 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914768    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:32:41.919427 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914770    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:32:41.919427 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914773    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:32:41.919427 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914775    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:32:41.919427 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914778    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:32:41.919427 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914781    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:32:41.919427 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914783    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:32:41.919427 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914786    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:32:41.919427 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914788    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:32:41.919427 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914791    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:32:41.919427 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914793    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:32:41.919427 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914796    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:32:41.919427 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914801    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:32:41.919427 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914805    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:32:41.919427 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914808    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:32:41.919427 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914811    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:32:41.919427 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914814    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:32:41.919427 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914816    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:32:41.919427 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914820    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:32:41.919427 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914823    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:32:41.919863 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914825    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:32:41.919863 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914828    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:32:41.919863 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914831    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:32:41.919863 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914834    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:32:41.919863 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914837    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:32:41.919863 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914840    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:32:41.919863 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914842    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:32:41.919863 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914845    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:32:41.919863 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914847    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:32:41.919863 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914851    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:32:41.919863 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914855    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:32:41.919863 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914859    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:32:41.919863 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914861    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:32:41.919863 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914864    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:32:41.919863 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914867    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:32:41.919863 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914869    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:32:41.919863 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914871    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:32:41.919863 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914874    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:32:41.919863 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914876    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:32:41.919863 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914879    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:32:41.920326 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914881    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:32:41.920326 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914884    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:32:41.920326 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914886    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:32:41.920326 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914889    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:32:41.920326 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914891    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:32:41.920326 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914894    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:32:41.920326 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914896    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:32:41.920326 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914898    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:32:41.920326 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914901    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:32:41.920326 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914903    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:32:41.920326 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914907    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:32:41.920326 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914910    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:32:41.920326 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914912    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:32:41.920326 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914915    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:32:41.920326 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914917    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:32:41.920326 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914920    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:32:41.920326 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914923    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:32:41.920326 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914926    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:32:41.920326 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914928    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:32:41.920775 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914930    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:32:41.920775 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914933    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:32:41.920775 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914936    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:32:41.920775 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914939    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:32:41.920775 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.914944    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:32:41.920775 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915865    2572 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:32:41.920775 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915876    2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:32:41.920775 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915883    2572 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:32:41.920775 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915888    2572 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:32:41.920775 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915892    2572 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:32:41.920775 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915895    2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:32:41.920775 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915903    2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:32:41.920775 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915907    2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:32:41.920775 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915910    2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:32:41.920775 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915913    2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:32:41.920775 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915916    2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:32:41.920775 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915920    2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:32:41.920775 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915923    2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:32:41.920775 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915926    2572 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:32:41.920775 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915929    2572 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:32:41.920775 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915932    2572 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:32:41.920775 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915935    2572 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:32:41.920775 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915937    2572 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:32:41.921286 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915942    2572 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:32:41.921286 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915945    2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:32:41.921286 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915948    2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:32:41.921286 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915951    2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:32:41.921286 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915954    2572 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:32:41.921286 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915959    2572 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:32:41.921286 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915962    2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:32:41.921286 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915966    2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:32:41.921286 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915969    2572 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:32:41.921286 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915972    2572 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:32:41.921286 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915975    2572 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:32:41.921286 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915977    2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:32:41.921286 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915981    2572 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:32:41.921286 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915984    2572 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:32:41.921286 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915988    2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:32:41.921286 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915991    2572 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:32:41.921286 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915994    2572 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:32:41.921286 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.915997    2572 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:32:41.921286 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916000    2572 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:32:41.921286 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916002    2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:32:41.921286 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916007    2572 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:32:41.921286 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916010    2572 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:32:41.921286 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916013    2572 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:32:41.921286 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916016    2572 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:32:41.921868 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916019    2572 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:32:41.921868 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916023    2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:32:41.921868 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916026    2572 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:32:41.921868 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916029    2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:32:41.921868 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916032    2572 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:32:41.921868 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916035    2572 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:32:41.921868 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916037    2572 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:32:41.921868 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916040    2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:32:41.921868 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916043    2572 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:32:41.921868 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916046    2572 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:32:41.921868 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916049    2572 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:32:41.921868 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916052    2572 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:32:41.921868 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916056    2572 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:32:41.921868 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916059    2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:32:41.921868 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916063    2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:32:41.921868 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916066    2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:32:41.921868 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916069    2572 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:32:41.921868 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916072    2572 flags.go:64] FLAG: --help="false"
Oct 30 21:32:41.921868 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916075    2572 flags.go:64] FLAG: --hostname-override="ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:32:41.921868 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916079    2572 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:32:41.921868 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916081    2572 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:32:41.921868 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916085    2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:32:41.921868 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916088    2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:32:41.922410 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916092    2572 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:32:41.922410 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916095    2572 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:32:41.922410 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916098    2572 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:32:41.922410 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916101    2572 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:32:41.922410 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916104    2572 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:32:41.922410 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916107    2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:32:41.922410 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916110    2572 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:32:41.922410 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916113    2572 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:32:41.922410 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916116    2572 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:32:41.922410 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916119    2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:32:41.922410 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916122    2572 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:32:41.922410 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916125    2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:32:41.922410 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916128    2572 flags.go:64] FLAG: --lock-file=""
Oct 30 21:32:41.922410 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916130    2572 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:32:41.922410 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916133    2572 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:32:41.922410 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916136    2572 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:32:41.922410 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916142    2572 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:32:41.922410 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916145    2572 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:32:41.922410 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916148    2572 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:32:41.922410 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916151    2572 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:32:41.922410 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916154    2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:32:41.922410 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916157    2572 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:32:41.922410 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916160    2572 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:32:41.922410 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916163    2572 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:32:41.922410 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916167    2572 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:32:41.923040 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916170    2572 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:32:41.923040 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916174    2572 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:32:41.923040 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916177    2572 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:32:41.923040 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916181    2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:32:41.923040 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916184    2572 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:32:41.923040 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916186    2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:32:41.923040 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916189    2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:32:41.923040 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916192    2572 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:32:41.923040 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916195    2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:32:41.923040 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916202    2572 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:32:41.923040 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916205    2572 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:32:41.923040 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916208    2572 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:32:41.923040 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916211    2572 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:32:41.923040 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916214    2572 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:32:41.923040 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916217    2572 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:32:41.923040 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916220    2572 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:32:41.923040 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916222    2572 flags.go:64] FLAG: --port="10250"
Oct 30 21:32:41.923040 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916225    2572 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:32:41.923040 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916229    2572 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-05c15d85dc3eaf985"
Oct 30 21:32:41.923040 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916232    2572 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:32:41.923040 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916235    2572 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:32:41.923040 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916238    2572 flags.go:64] FLAG: --register-node="true"
Oct 30 21:32:41.923040 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916241    2572 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:32:41.923040 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916244    2572 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:32:41.923040 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916248    2572 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:32:41.923040 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916250    2572 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:32:41.923713 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916253    2572 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:32:41.923713 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916257    2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:32:41.923713 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916260    2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:32:41.923713 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916263    2572 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:32:41.923713 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916266    2572 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:32:41.923713 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916269    2572 flags.go:64] FLAG: --runonce="false"
Oct 30 21:32:41.923713 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916271    2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:32:41.923713 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916274    2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:32:41.923713 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916278    2572 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:32:41.923713 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916281    2572 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:32:41.923713 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916284    2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:32:41.923713 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916287    2572 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:32:41.923713 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916290    2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:32:41.923713 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916294    2572 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:32:41.923713 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916296    2572 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:32:41.923713 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916299    2572 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:32:41.923713 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916302    2572 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:32:41.923713 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916305    2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:32:41.923713 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916308    2572 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:32:41.923713 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916311    2572 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:32:41.923713 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916314    2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:32:41.923713 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916319    2572 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:32:41.923713 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916322    2572 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:32:41.923713 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916325    2572 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:32:41.923713 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916328    2572 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:32:41.924270 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916331    2572 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:32:41.924270 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916334    2572 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:32:41.924270 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916337    2572 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:32:41.924270 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916340    2572 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:32:41.924270 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916345    2572 flags.go:64] FLAG: --v="2"
Oct 30 21:32:41.924270 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916350    2572 flags.go:64] FLAG: --version="false"
Oct 30 21:32:41.924270 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916354    2572 flags.go:64] FLAG: --vmodule=""
Oct 30 21:32:41.924270 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916358    2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:32:41.924270 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916361    2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:32:41.924270 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916528    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:32:41.924270 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916533    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:32:41.924270 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916537    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:32:41.924270 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916540    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:32:41.924270 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916543    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:32:41.924270 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916546    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:32:41.924270 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916548    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:32:41.924270 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916552    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:32:41.924270 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916555    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:32:41.924270 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916557    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:32:41.924270 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916560    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:32:41.924270 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916563    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:32:41.924270 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916566    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:32:41.924901 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916568    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:32:41.924901 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916571    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:32:41.924901 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916573    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:32:41.924901 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916576    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:32:41.924901 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916579    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:32:41.924901 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916582    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:32:41.924901 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916585    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:32:41.924901 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916587    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:32:41.924901 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916590    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:32:41.924901 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916593    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:32:41.924901 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916595    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:32:41.924901 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916598    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:32:41.924901 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916601    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:32:41.924901 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916603    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:32:41.924901 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916607    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:32:41.924901 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916610    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:32:41.924901 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916612    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:32:41.924901 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916615    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:32:41.924901 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916618    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:32:41.925363 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916622    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:32:41.925363 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916624    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:32:41.925363 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916629    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:32:41.925363 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916632    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:32:41.925363 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916634    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:32:41.925363 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916637    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:32:41.925363 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916640    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:32:41.925363 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916642    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:32:41.925363 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916645    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:32:41.925363 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916648    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:32:41.925363 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916650    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:32:41.925363 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916653    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:32:41.925363 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916655    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:32:41.925363 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916658    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:32:41.925363 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916660    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:32:41.925363 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916663    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:32:41.925363 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916665    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:32:41.925363 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916668    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:32:41.925363 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916672    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:32:41.925856 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916675    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:32:41.925856 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916678    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:32:41.925856 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916681    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:32:41.925856 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916684    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:32:41.925856 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916687    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:32:41.925856 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916689    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:32:41.925856 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916692    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:32:41.925856 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916694    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:32:41.925856 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916698    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:32:41.925856 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916701    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:32:41.925856 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916703    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:32:41.925856 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916706    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:32:41.925856 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916708    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:32:41.925856 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916712    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:32:41.925856 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916715    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:32:41.925856 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916717    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:32:41.925856 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916720    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:32:41.925856 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916722    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:32:41.925856 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916725    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:32:41.926288 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916728    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:32:41.926288 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916730    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:32:41.926288 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916733    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:32:41.926288 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916736    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:32:41.926288 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916739    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:32:41.926288 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916741    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:32:41.926288 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916744    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:32:41.926288 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916746    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:32:41.926288 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916749    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:32:41.926288 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916752    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:32:41.926288 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916754    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:32:41.926288 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916756    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:32:41.926288 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916759    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:32:41.926288 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916761    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:32:41.926288 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916764    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:32:41.926288 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916766    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:32:41.926288 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916769    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:32:41.926288 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916771    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:32:41.926288 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916774    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:32:41.926288 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916776    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:32:41.926816 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916779    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:32:41.926816 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916783    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:32:41.926816 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916786    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:32:41.926816 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916788    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:32:41.926816 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916791    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:32:41.926816 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916794    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:32:41.926816 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916797    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:32:41.926816 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916800    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:32:41.926816 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916802    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:32:41.926816 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916805    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:32:41.926816 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916808    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:32:41.926816 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916810    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:32:41.926816 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916813    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:32:41.926816 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916815    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:32:41.926816 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916818    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:32:41.926816 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916820    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:32:41.927179 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.916845    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:32:41.927179 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916880    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:32:41.927179 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916885    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:32:41.927179 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916888    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:32:41.927179 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916891    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:32:41.927179 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916893    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:32:41.927179 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916896    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:32:41.927179 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916898    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:32:41.927179 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916901    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:32:41.927179 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916903    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:32:41.927179 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916906    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:32:41.927179 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916909    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:32:41.927179 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916911    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:32:41.927179 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916914    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:32:41.927179 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916917    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:32:41.927179 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916919    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:32:41.927572 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916922    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:32:41.927572 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916924    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:32:41.927572 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916928    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:32:41.927572 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916931    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:32:41.927572 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916933    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:32:41.927572 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916936    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:32:41.927572 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916940    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:32:41.927572 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916942    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:32:41.927572 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916945    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:32:41.927572 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916947    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:32:41.927572 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916950    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:32:41.927572 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916953    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:32:41.927572 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916955    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:32:41.927572 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916958    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:32:41.927572 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916960    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:32:41.927572 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916963    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:32:41.927572 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916966    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:32:41.927572 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916969    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:32:41.927572 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916971    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:32:41.927572 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916974    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:32:41.928038 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916977    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:32:41.928038 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916979    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:32:41.928038 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916982    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:32:41.928038 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916984    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:32:41.928038 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916987    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:32:41.928038 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916989    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:32:41.928038 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916992    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:32:41.928038 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916994    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:32:41.928038 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916997    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:32:41.928038 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.916999    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:32:41.928038 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917002    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:32:41.928038 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917005    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:32:41.928038 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917007    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:32:41.928038 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917009    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:32:41.928038 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917013    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:32:41.928038 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917016    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:32:41.928038 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917018    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:32:41.928038 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917022    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:32:41.928038 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917026    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:32:41.928494 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917029    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:32:41.928494 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917031    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:32:41.928494 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917034    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:32:41.928494 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917036    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:32:41.928494 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917039    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:32:41.928494 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917041    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:32:41.928494 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917044    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:32:41.928494 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917048    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:32:41.928494 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917053    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:32:41.928494 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917056    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:32:41.928494 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917059    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:32:41.928494 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917062    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:32:41.928494 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917065    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:32:41.928494 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917068    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:32:41.928494 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917070    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:32:41.928494 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917073    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:32:41.928494 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917075    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:32:41.928494 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917078    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:32:41.928494 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917080    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:32:41.928934 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917083    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:32:41.928934 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917085    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:32:41.928934 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917088    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:32:41.928934 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917090    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:32:41.928934 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917094    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:32:41.928934 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917097    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:32:41.928934 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917100    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:32:41.928934 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917103    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:32:41.928934 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917107    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:32:41.928934 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917109    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:32:41.928934 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917112    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:32:41.928934 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917114    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:32:41.928934 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917118    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:32:41.928934 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917121    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:32:41.928934 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917123    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:32:41.928934 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917126    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:32:41.928934 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917129    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:32:41.928934 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917131    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:32:41.928934 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917134    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:32:41.929361 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917136    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:32:41.929361 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917139    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:32:41.929361 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917141    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:32:41.929361 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917144    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:32:41.929361 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917147    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:32:41.929361 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917149    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:32:41.929361 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917152    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:32:41.929361 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917155    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:32:41.929361 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917157    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:32:41.929361 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917162    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:32:41.929361 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917164    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:32:41.929361 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917167    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:32:41.929361 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917169    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:32:41.929361 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.917172    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:32:41.929361 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.924283    2572 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:32:41.929361 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.924297    2572 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:32:41.929361 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.924313    2572 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:32:41.929361 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.924317    2572 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:32:41.929361 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924375    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:32:41.929361 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924381    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:32:41.929834 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924384    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:32:41.929834 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924387    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:32:41.929834 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924390    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:32:41.929834 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924393    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:32:41.929834 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924396    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:32:41.929834 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924400    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:32:41.929834 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924405    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:32:41.929834 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924408    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:32:41.929834 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924410    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:32:41.929834 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924428    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:32:41.929834 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924431    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:32:41.929834 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924434    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:32:41.929834 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924436    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:32:41.929834 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924439    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:32:41.929834 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924442    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:32:41.929834 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924445    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:32:41.929834 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924447    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:32:41.929834 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924450    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:32:41.929834 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924453    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:32:41.930272 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924456    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:32:41.930272 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924459    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:32:41.930272 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924462    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:32:41.930272 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924465    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:32:41.930272 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924467    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:32:41.930272 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924470    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:32:41.930272 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924473    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:32:41.930272 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924475    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:32:41.930272 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924480    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:32:41.930272 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924484    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:32:41.930272 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924487    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:32:41.930272 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924490    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:32:41.930272 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924493    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:32:41.930272 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924496    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:32:41.930272 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924499    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:32:41.930272 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924501    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:32:41.930272 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924504    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:32:41.930272 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924506    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:32:41.930272 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924509    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:32:41.930715 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924511    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:32:41.930715 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924514    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:32:41.930715 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924516    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:32:41.930715 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924519    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:32:41.930715 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924521    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:32:41.930715 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924524    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:32:41.930715 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924526    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:32:41.930715 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924529    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:32:41.930715 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924532    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:32:41.930715 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924534    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:32:41.930715 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924537    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:32:41.930715 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924539    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:32:41.930715 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924542    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:32:41.930715 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924545    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:32:41.930715 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924548    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:32:41.930715 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924550    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:32:41.930715 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924553    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:32:41.930715 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924555    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:32:41.930715 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924558    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:32:41.930715 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924561    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:32:41.931250 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924563    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:32:41.931250 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924566    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:32:41.931250 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924569    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:32:41.931250 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924572    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:32:41.931250 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924574    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:32:41.931250 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924577    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:32:41.931250 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924580    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:32:41.931250 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924582    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:32:41.931250 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924585    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:32:41.931250 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924588    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:32:41.931250 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924591    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:32:41.931250 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924593    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:32:41.931250 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924595    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:32:41.931250 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924598    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:32:41.931250 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924600    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:32:41.931250 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924603    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:32:41.931250 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924605    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:32:41.931250 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924608    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:32:41.931250 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924610    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:32:41.931250 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924613    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:32:41.931772 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924616    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:32:41.931772 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924618    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:32:41.931772 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924621    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:32:41.931772 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924623    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:32:41.931772 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924626    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:32:41.931772 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924629    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:32:41.931772 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924632    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:32:41.931772 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924635    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:32:41.931772 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924637    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:32:41.931772 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924639    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:32:41.931772 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924642    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:32:41.931772 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924644    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:32:41.931772 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924647    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:32:41.931772 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924650    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:32:41.931772 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924653    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:32:41.931772 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924655    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:32:41.931772 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924658    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:32:41.931772 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924660    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:32:41.931772 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924663    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:32:41.932205 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924665    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:32:41.932205 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924668    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:32:41.932205 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924671    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:32:41.932205 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924673    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:32:41.932205 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924675    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:32:41.932205 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924678    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:32:41.932205 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924680    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:32:41.932205 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.924707    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:32:41.932205 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924843    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:32:41.932205 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924848    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:32:41.932205 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924852    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:32:41.932205 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924856    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:32:41.932205 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924860    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:32:41.932205 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924863    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:32:41.932205 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924866    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:32:41.932205 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924868    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:32:41.932594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924871    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:32:41.932594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924874    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:32:41.932594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924878    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:32:41.932594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924880    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:32:41.932594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924884    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:32:41.932594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924886    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:32:41.932594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924889    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:32:41.932594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924892    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:32:41.932594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924894    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:32:41.932594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924896    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:32:41.932594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924899    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:32:41.932594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924902    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:32:41.932594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924905    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:32:41.932594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924907    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:32:41.932594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924910    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:32:41.932594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924913    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:32:41.932594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924915    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:32:41.932594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924918    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:32:41.932594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924921    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:32:41.933021 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924923    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:32:41.933021 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924925    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:32:41.933021 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924928    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:32:41.933021 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924930    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:32:41.933021 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924933    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:32:41.933021 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924935    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:32:41.933021 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924938    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:32:41.933021 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924940    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:32:41.933021 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924943    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:32:41.933021 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924945    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:32:41.933021 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924948    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:32:41.933021 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924950    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:32:41.933021 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924953    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:32:41.933021 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924955    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:32:41.933021 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924957    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:32:41.933021 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924960    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:32:41.933021 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924962    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:32:41.933021 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924965    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:32:41.933021 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924968    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:32:41.933021 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924970    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:32:41.933594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924973    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:32:41.933594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924975    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:32:41.933594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924978    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:32:41.933594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924980    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:32:41.933594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924983    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:32:41.933594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924985    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:32:41.933594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924988    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:32:41.933594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924990    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:32:41.933594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924993    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:32:41.933594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924995    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:32:41.933594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.924997    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:32:41.933594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925000    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:32:41.933594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925002    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:32:41.933594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925004    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:32:41.933594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925007    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:32:41.933594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925010    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:32:41.933594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925014    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:32:41.933594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925017    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:32:41.933594 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925020    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:32:41.934047 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925023    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:32:41.934047 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925026    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:32:41.934047 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925028    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:32:41.934047 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925031    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:32:41.934047 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925034    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:32:41.934047 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925036    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:32:41.934047 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925039    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:32:41.934047 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925041    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:32:41.934047 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925044    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:32:41.934047 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925046    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:32:41.934047 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925049    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:32:41.934047 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925052    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:32:41.934047 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925055    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:32:41.934047 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925058    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:32:41.934047 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925060    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:32:41.934047 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925063    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:32:41.934047 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925065    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:32:41.934047 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925067    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:32:41.934047 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925070    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:32:41.934555 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925072    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:32:41.934555 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925075    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:32:41.934555 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925077    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:32:41.934555 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925080    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:32:41.934555 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925082    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:32:41.934555 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925084    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:32:41.934555 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925087    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:32:41.934555 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925089    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:32:41.934555 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925092    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:32:41.934555 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925094    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:32:41.934555 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925097    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:32:41.934555 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925099    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:32:41.934555 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925102    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:32:41.934555 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925104    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:32:41.934555 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925107    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:32:41.934555 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925109    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:32:41.934555 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925112    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:32:41.934555 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925114    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:32:41.934555 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925117    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:32:41.934555 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925119    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:32:41.935120 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925121    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:32:41.935120 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.925143    2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:32:41.935120 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925175    2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:32:41.935120 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925179    2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:32:41.935120 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925182    2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:32:41.935120 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925184    2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:32:41.935120 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925187    2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:32:41.935120 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925190    2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:32:41.935120 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925193    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:32:41.935120 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925195    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:32:41.935120 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925198    2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:32:41.935120 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925200    2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:32:41.935120 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925202    2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:32:41.935120 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925205    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:32:41.935120 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925207    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:32:41.935538 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925210    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:32:41.935538 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925212    2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:32:41.935538 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925215    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:32:41.935538 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925217    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:32:41.935538 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925220    2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:32:41.935538 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925222    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:32:41.935538 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925224    2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:32:41.935538 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925227    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:32:41.935538 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925229    2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:32:41.935538 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925232    2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:32:41.935538 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925235    2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:32:41.935538 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925237    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:32:41.935538 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925240    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:32:41.935538 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925242    2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:32:41.935538 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925245    2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:32:41.935538 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925248    2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:32:41.935538 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925250    2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:32:41.935538 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925253    2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:32:41.935538 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925255    2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:32:41.935538 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925257    2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:32:41.936340 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925260    2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:32:41.936340 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925262    2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:32:41.936340 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925265    2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:32:41.936340 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925267    2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:32:41.936340 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925270    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:32:41.936340 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925273    2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:32:41.936340 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925275    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:32:41.936340 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925278    2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:32:41.936340 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925280    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:32:41.936340 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925283    2572 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:32:41.936340 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925285    2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:32:41.936340 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925287    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:32:41.936340 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925290    2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:32:41.936340 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925292    2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:32:41.936340 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925295    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:32:41.936340 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925297    2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:32:41.936340 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925301    2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:32:41.936340 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925305    2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:32:41.936340 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925308    2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:32:41.936802 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925310    2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:32:41.936802 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925313    2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:32:41.936802 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925315    2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:32:41.936802 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925318    2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:32:41.936802 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925320    2572 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:32:41.936802 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925323    2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:32:41.936802 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925325    2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:32:41.936802 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925328    2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:32:41.936802 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925330    2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:32:41.936802 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925333    2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:32:41.936802 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925335    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:32:41.936802 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925338    2572 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:32:41.936802 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925340    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:32:41.936802 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925343    2572 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:32:41.936802 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925346    2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:32:41.936802 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925348    2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:32:41.936802 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925351    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:32:41.936802 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925353    2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:32:41.936802 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925356    2572 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:32:41.937310 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925359    2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:32:41.937310 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925361    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:32:41.937310 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925364    2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:32:41.937310 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925367    2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:32:41.937310 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925369    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:32:41.937310 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925372    2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:32:41.937310 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925374    2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:32:41.937310 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925377    2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:32:41.937310 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925380    2572 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:32:41.937310 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925382    2572 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:32:41.937310 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925385    2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:32:41.937310 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925387    2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:32:41.937310 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925389    2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:32:41.937310 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925392    2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:32:41.937310 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925395    2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:32:41.937310 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925397    2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:32:41.937310 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925400    2572 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:32:41.937310 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925402    2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:32:41.937310 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925405    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:32:41.937310 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925408    2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:32:41.937961 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925410    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:32:41.937961 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925425    2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:32:41.937961 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925427    2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:32:41.937961 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925431    2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:32:41.937961 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925434    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:32:41.937961 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925437    2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:32:41.937961 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925440    2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:32:41.937961 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925443    2572 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:32:41.937961 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925445    2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:32:41.937961 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925448    2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:32:41.937961 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925452    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:32:41.937961 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925454    2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:32:41.937961 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925457    2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:32:41.937961 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925460    2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:32:41.937961 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:41.925462    2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:32:41.937961 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.925511    2572 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:32:41.937961 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.927763    2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 21:32:41.937961 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.928672    2572 server.go:1014] "Starting client certificate rotation"
Oct 30 21:32:41.937961 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.928770    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:32:41.938461 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.928799    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:32:41.938461 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.931605    2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:32:41.938461 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.937820    2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:32:41.947139 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.947121    2572 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:32:41.951590 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.951574    2572 log.go:25] "Validated CRI v1 image API"
Oct 30 21:32:41.952379 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.952353    2572 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:32:41.956667 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.956642    2572 fs.go:135] Filesystem UUIDs: map[248b64a8-55af-4ae5-8745-847a977ba628:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fa2af845-2974-4011-b9d5-793eb5e1d844:/dev/nvme0n1p4]
Oct 30 21:32:41.956711 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.956668    2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 21:32:41.961903 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.961793    2572 manager.go:217] Machine: {Timestamp:2026-10-30 21:32:41.96032079 +0000 UTC m=+0.343217354 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099738 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a4c1129a9e55f9492dc1481817b00 SystemUUID:ec2a4c11-29a9-e55f-9492-dc1481817b00 BootID:3a636b23-aa57-4caf-a687-6f46766e2cdc Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:44:af:0d:e0:7b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:44:af:0d:e0:7b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:8a:f6:0a:4f:3b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:32:41.961903 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.961902    2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:32:41.962030 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.962019    2572 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:32:41.964734 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.964713    2572 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:32:41.964881 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.964735    2572 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-92.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:32:41.964921 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.964884    2572 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:32:41.964921 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.964891    2572 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:32:41.964921 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.964917    2572 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:32:41.965006 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.964932    2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:32:41.966164 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.966153    2572 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:32:41.970528 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.970516    2572 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:32:41.970655 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.970646    2572 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:32:41.971463 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.971454    2572 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:32:41.971501 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.971467    2572 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:32:41.971501 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.971478    2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:32:41.971501 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.971486    2572 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:32:41.971501 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.971496    2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:32:41.971832 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.971817    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:32:41.974228 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.974203    2572 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:32:41.980681 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.980661    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:32:41.981626 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.981612    2572 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:32:41.981702 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.981660    2572 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:32:41.981852 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.981839    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:32:41.981910 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.981859    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:32:41.981910 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.981869    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:32:41.981910 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.981878    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:32:41.981910 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.981886    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:32:41.981910 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.981895    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:32:41.982677 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.982667    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:32:41.982726 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.982682    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:32:41.982726 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.982692    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:32:41.982726 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.982703    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:32:41.982726 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.982717    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:32:41.982945 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.982731    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:32:41.983485 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.983474    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:32:41.983544 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.983489    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:32:41.990050 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.990035    2572 server.go:1263] "Started kubelet"
Oct 30 21:32:41.990126 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.990103    2572 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:32:41.990188 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.990128    2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:32:41.990241 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.990191    2572 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:32:41.990718 ip-10-0-132-92 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:32:41.991926 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.991906    2572 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:32:41.992575 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.992552    2572 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:32:41.995595 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:41.995574    2572 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 21:32:41.999969 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.999956    2572 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:32:42.000027 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:41.999980    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:32:42.000076 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:42.000039    2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-92.us-west-1.compute.internal\" not found"
Oct 30 21:32:42.000758 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.000743    2572 factory.go:55] Registering systemd factory
Oct 30 21:32:42.000848 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.000784    2572 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:32:42.000848 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.000830    2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:32:42.000986 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.000830    2572 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:32:42.000986 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.000866    2572 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:32:42.000986 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.000945    2572 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:32:42.000986 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.000952    2572 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:32:42.001140 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.000994    2572 factory.go:153] Registering CRI-O factory
Oct 30 21:32:42.001140 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.001027    2572 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:32:42.001140 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.001110    2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:32:42.001140 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.001131    2572 factory.go:103] Registering Raw factory
Oct 30 21:32:42.001268 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.001155    2572 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:32:42.002152 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.002129    2572 manager.go:319] Starting recovery of all containers
Oct 30 21:32:42.013660 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.013024    2572 manager.go:324] Recovery completed
Oct 30 21:32:42.015100 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:42.015080    2572 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Oct 30 21:32:42.017966 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.017953    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:32:42.019538 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.019523    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:32:42.019605 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.019554    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:32:42.019605 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.019566    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:32:42.020013 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.019999    2572 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:32:42.020077 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.020013    2572 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:32:42.020077 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.020032    2572 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:32:42.020077 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.020059    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:32:42.020721 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.020707    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:32:42.020785 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.020738    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:32:42.020785 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.020748    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:32:42.022206 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.022193    2572 policy_none.go:50] "Start"
Oct 30 21:32:42.022264 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.022210    2572 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:32:42.022264 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.022224    2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:32:42.024349 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.024336    2572 policy_none.go:44] "Start"
Oct 30 21:32:42.085738 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.072941    2572 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:32:42.085738 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.077098    2572 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:32:42.085738 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:42.077127    2572 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:32:42.085738 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.077136    2572 server.go:87] "Starting device plugin registration server"
Oct 30 21:32:42.085738 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.077332    2572 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:32:42.085738 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.077342    2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:32:42.085738 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.077555    2572 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:32:42.085738 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.077625    2572 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:32:42.085738 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.077631    2572 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:32:42.085738 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:42.078580    2572 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:32:42.085738 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:42.078636    2572 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-92.us-west-1.compute.internal\" not found"
Oct 30 21:32:42.085738 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.084271    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:32:42.085738 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.085553    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:32:42.085738 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.085571    2572 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:32:42.085738 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.085587    2572 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:32:42.085738 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:42.085622    2572 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 21:32:42.131863 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.131842    2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-92.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 21:32:42.131863 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:42.131849    2572 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-92.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 21:32:42.132963 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:42.131846    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-92.us-west-1.compute.internal.18e36c202808c0f7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-92.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-92.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:32:41.990004983 +0000 UTC m=+0.372901546,LastTimestamp:2026-10-30 21:32:41.990004983 +0000 UTC m=+0.372901546,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-92.us-west-1.compute.internal,}"
Oct 30 21:32:42.178001 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.177952    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:32:42.178757 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.178743    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:32:42.178802 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.178770    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:32:42.178802 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.178782    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:32:42.178866 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.178809    2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:32:42.186002 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.185979    2572 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal"]
Oct 30 21:32:42.186100 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.186045    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:32:42.186808 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.186792    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:32:42.186878 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.186819    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:32:42.186878 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.186839    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:32:42.188882 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.188871    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:32:42.189043 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.189030    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:32:42.189081 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.189060    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:32:42.189657 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.189638    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:32:42.189740 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.189675    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:32:42.189740 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.189689    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:32:42.189806 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.189677    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:32:42.189806 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.189773    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:32:42.189806 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.189794    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:32:42.191791 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.191773    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:32:42.191863 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.191808    2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:32:42.192505 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.192492    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:32:42.192560 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.192513    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:32:42.192560 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.192528    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:32:42.201998 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.201982    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/04840b9eb1acb1deca2b80fddf33a1d0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal\" (UID: \"04840b9eb1acb1deca2b80fddf33a1d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:32:42.202081 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.202013    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04840b9eb1acb1deca2b80fddf33a1d0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal\" (UID: \"04840b9eb1acb1deca2b80fddf33a1d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:32:42.202081 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.202041    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57c3091500d45dc621d6409761c7c97d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal\" (UID: \"57c3091500d45dc621d6409761c7c97d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:32:42.204638 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.204623    2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:32:42.204696 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.204660    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dl98z"
Oct 30 21:32:42.212931 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:42.212905    2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-92.us-west-1.compute.internal\" not found" node="ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:32:42.214965 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:42.214950    2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-92.us-west-1.compute.internal\" not found" node="ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:32:42.287937 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.287912    2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:32:42.289457 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.289442    2572 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:32:42.303189 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.303169    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/04840b9eb1acb1deca2b80fddf33a1d0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal\" (UID: \"04840b9eb1acb1deca2b80fddf33a1d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:32:42.303262 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.303203    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/04840b9eb1acb1deca2b80fddf33a1d0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal\" (UID: \"04840b9eb1acb1deca2b80fddf33a1d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:32:42.303262 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.303224    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04840b9eb1acb1deca2b80fddf33a1d0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal\" (UID: \"04840b9eb1acb1deca2b80fddf33a1d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:32:42.303262 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.303240    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57c3091500d45dc621d6409761c7c97d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal\" (UID: \"57c3091500d45dc621d6409761c7c97d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:32:42.303262 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.303257    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57c3091500d45dc621d6409761c7c97d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal\" (UID: \"57c3091500d45dc621d6409761c7c97d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:32:42.303392 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.303277    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04840b9eb1acb1deca2b80fddf33a1d0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal\" (UID: \"04840b9eb1acb1deca2b80fddf33a1d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:32:42.387294 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.387270    2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:32:42.389441 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.389406    2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:32:42.390141 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.390109    2572 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:32:42.390141 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:42.390138    2572 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-92.us-west-1.compute.internal\": node \"ip-10-0-132-92.us-west-1.compute.internal\" not found"
Oct 30 21:32:42.400157 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.400138    2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:32:42.508655 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.508600    2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:32:42.508733 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.508689    2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:32:42.517745 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.517727    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:32:42.520888 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.520875    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:32:42.563970 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.563951    2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:32:42.928958 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.928876    2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 21:32:42.929479 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.928994    2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 21:32:42.929479 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.928996    2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 21:32:42.929479 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:42.929027    2572 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a87acbcee0c914c0587855f827b89858-cda6e39977984c9a.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-132-92.us-west-1.compute.internal.18e36c2029cbe3af\": read tcp 10.0.132.92:37614->18.190.185.126:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-132-92.us-west-1.compute.internal.18e36c2029cbe3af  default   4832 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-92.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-132-92.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-132-92.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 21:32:42 +0000 UTC,LastTimestamp:2026-10-30 21:32:42.178788299 +0000 UTC m=+0.561684863,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-92.us-west-1.compute.internal,}"
Oct 30 21:32:42.971943 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:42.971924    2572 apiserver.go:52] "Watching apiserver"
Oct 30 21:32:43.000506 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.000484    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 21:32:43.209656 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.209575    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:27:43 +0000 UTC" deadline="2028-04-12 14:03:38.413170341 +0000 UTC"
Oct 30 21:32:43.209656 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.209601    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12712h30m55.20357194s"
Oct 30 21:32:43.232665 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.232635    2572 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:32:43.232979 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.232959    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-h5xkl","openshift-multus/multus-25lhq","openshift-multus/multus-additional-cni-plugins-956lw","openshift-network-diagnostics/network-check-target-spx9v","openshift-dns/node-resolver-9xqln","openshift-image-registry/node-ca-hnmxk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal","openshift-network-operator/iptables-alerter-w9ckc","openshift-ovn-kubernetes/ovnkube-node-bjt99","kube-system/global-pull-secret-syncer-2z8sh","kube-system/kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-86426","openshift-multus/network-metrics-daemon-bxbq6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"]
Oct 30 21:32:43.235370 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.235348    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h5xkl"
Oct 30 21:32:43.237437 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.237401    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.239472 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.239452    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:32:43.241469 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.241451    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:32:43.241554 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:43.241514    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-spx9v" podUID="e1a448f1-e180-4fbf-9923-fac0b9dbec74"
Oct 30 21:32:43.243610 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.243586    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9xqln"
Oct 30 21:32:43.245682 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.245660    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hnmxk"
Oct 30 21:32:43.247766 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.247750    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w9ckc"
Oct 30 21:32:43.250016 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.250002    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.253106 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.253089    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:32:43.253196 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:43.253168    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2z8sh" podUID="2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49"
Oct 30 21:32:43.255917 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.255897    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.258103 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.258086    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:32:43.258169 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:43.258141    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxbq6" podUID="032b9335-657e-4a86-aa6d-4ef46163014a"
Oct 30 21:32:43.260276 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.260261    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:32:43.288814 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.288793    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:32:43.288970 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.288949    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-wn5fv\""
Oct 30 21:32:43.289033 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.288986    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:32:43.289363 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.289343    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:32:43.289559 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.289542    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:32:43.289615 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.289570    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:32:43.289615 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.289586    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:32:43.289615 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.289599    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-zs2mb\""
Oct 30 21:32:43.290608 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.290593    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:32:43.290665 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.290591    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:32:43.290704 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.290664    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rjmnn\""
Oct 30 21:32:43.294889 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.294874    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:32:43.294964 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.294923    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mzgn4\""
Oct 30 21:32:43.295001 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.294977    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:32:43.296933 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.296917    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:32:43.296933 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.296929    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2v7jg\""
Oct 30 21:32:43.297021 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.296955    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:32:43.297021 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.297003    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:32:43.301757 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.301743    2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:32:43.309519 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.309505    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-run-multus-certs\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.309575 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.309527    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkvjz\" (UniqueName: \"kubernetes.io/projected/9a1460f5-20f5-4bd1-843f-3838935f2ce2-kube-api-access-lkvjz\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.309575 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.309543    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcz4j\" (UniqueName: \"kubernetes.io/projected/e1a448f1-e180-4fbf-9923-fac0b9dbec74-kube-api-access-xcz4j\") pod \"network-check-target-spx9v\" (UID: \"e1a448f1-e180-4fbf-9923-fac0b9dbec74\") " pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:32:43.309575 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.309557    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-run-systemd\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.309575 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.309573    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs\") pod \"network-metrics-daemon-bxbq6\" (UID: \"032b9335-657e-4a86-aa6d-4ef46163014a\") " pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:32:43.309706 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.309620    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:32:43.309706 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.309644    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-slash\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.309706 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.309684    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-original-pull-secret\") pod \"global-pull-secret-syncer-2z8sh\" (UID: \"2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49\") " pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:32:43.309847 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.309734    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-socket-dir\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:32:43.309847 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.309791    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-registration-dir\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:32:43.309847 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.309819    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a1460f5-20f5-4bd1-843f-3838935f2ce2-cni-binary-copy\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.309945 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.309860    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-multus-socket-dir-parent\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.309945 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.309914    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-run-k8s-cni-cncf-io\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.310035 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.309943    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a20999e6-4f67-47b9-9cf1-da1b1fb027f2-host\") pod \"node-ca-hnmxk\" (UID: \"a20999e6-4f67-47b9-9cf1-da1b1fb027f2\") " pod="openshift-image-registry/node-ca-hnmxk"
Oct 30 21:32:43.310035 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.309970    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-var-lib-openvswitch\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.310035 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310024    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-etc-openvswitch\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.310164 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310056    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgwnq\" (UniqueName: \"kubernetes.io/projected/7986120b-482f-4b0b-a605-32770569962d-kube-api-access-pgwnq\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.310164 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310074    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-run\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.310164 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310103    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/752a05d6-f630-42ec-9795-0d28e9ed7d80-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:32:43.310164 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310136    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/061313cb-4d50-4ae5-afda-dc63a62a013b-tmp\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.310349 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310177    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rbpk\" (UniqueName: \"kubernetes.io/projected/23a040b8-2420-4d59-b426-f52832dc7de1-kube-api-access-7rbpk\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:32:43.310349 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310193    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/239a77e8-635c-4388-ad79-1cb3b15129d8-tmp-dir\") pod \"node-resolver-9xqln\" (UID: \"239a77e8-635c-4388-ad79-1cb3b15129d8\") " pod="openshift-dns/node-resolver-9xqln"
Oct 30 21:32:43.310349 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310207    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-run-netns\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.310349 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310226    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-multus-conf-dir\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.310349 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310275    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-log-socket\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.310349 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310304    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-dbus\") pod \"global-pull-secret-syncer-2z8sh\" (UID: \"2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49\") " pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:32:43.310349 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310341    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/239a77e8-635c-4388-ad79-1cb3b15129d8-hosts-file\") pod \"node-resolver-9xqln\" (UID: \"239a77e8-635c-4388-ad79-1cb3b15129d8\") " pod="openshift-dns/node-resolver-9xqln"
Oct 30 21:32:43.310623 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310371    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t68x5\" (UniqueName: \"kubernetes.io/projected/239a77e8-635c-4388-ad79-1cb3b15129d8-kube-api-access-t68x5\") pod \"node-resolver-9xqln\" (UID: \"239a77e8-635c-4388-ad79-1cb3b15129d8\") " pod="openshift-dns/node-resolver-9xqln"
Oct 30 21:32:43.310623 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310392    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-sysctl-d\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.310623 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310435    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdkhn\" (UniqueName: \"kubernetes.io/projected/061313cb-4d50-4ae5-afda-dc63a62a013b-kube-api-access-qdkhn\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.310623 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310457    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-run-netns\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.310623 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310473    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-cni-bin\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.310623 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310487    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-sysconfig\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.310623 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310515    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-sysctl-conf\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.310623 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310541    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-device-dir\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:32:43.310623 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310566    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/752a05d6-f630-42ec-9795-0d28e9ed7d80-cni-binary-copy\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:32:43.310623 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310580    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-run-ovn\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.310623 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310597    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-kubernetes\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.311017 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310627    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-etc-kubernetes\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.311017 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310717    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/752a05d6-f630-42ec-9795-0d28e9ed7d80-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:32:43.311017 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310737    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a7ef1f3-4e9d-40db-93a3-4d38a3474e89-host-slash\") pod \"iptables-alerter-w9ckc\" (UID: \"2a7ef1f3-4e9d-40db-93a3-4d38a3474e89\") " pod="openshift-network-operator/iptables-alerter-w9ckc"
Oct 30 21:32:43.311017 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310753    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-kubelet-config\") pod \"global-pull-secret-syncer-2z8sh\" (UID: \"2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49\") " pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:32:43.311017 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310793    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-var-lib-kubelet\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.311017 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310822    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2a7ef1f3-4e9d-40db-93a3-4d38a3474e89-iptables-alerter-script\") pod \"iptables-alerter-w9ckc\" (UID: \"2a7ef1f3-4e9d-40db-93a3-4d38a3474e89\") " pod="openshift-network-operator/iptables-alerter-w9ckc"
Oct 30 21:32:43.311017 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310846    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-run-ovn-kubernetes\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.311017 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310873    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.311017 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310897    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-multus-cni-dir\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.311017 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310925    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-var-lib-cni-bin\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.311017 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310945    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/752a05d6-f630-42ec-9795-0d28e9ed7d80-system-cni-dir\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:32:43.311017 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310969    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/752a05d6-f630-42ec-9795-0d28e9ed7d80-tuning-conf-dir\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:32:43.311017 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.310985    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkjl7\" (UniqueName: \"kubernetes.io/projected/a20999e6-4f67-47b9-9cf1-da1b1fb027f2-kube-api-access-lkjl7\") pod \"node-ca-hnmxk\" (UID: \"a20999e6-4f67-47b9-9cf1-da1b1fb027f2\") " pod="openshift-image-registry/node-ca-hnmxk"
Oct 30 21:32:43.311017 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.311015    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7986120b-482f-4b0b-a605-32770569962d-env-overrides\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.311582 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.311060    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-lib-modules\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.311582 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.311086    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-tuned\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.311582 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.311116    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9a1460f5-20f5-4bd1-843f-3838935f2ce2-multus-daemon-config\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.311582 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.311149    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwsj7\" (UniqueName: \"kubernetes.io/projected/752a05d6-f630-42ec-9795-0d28e9ed7d80-kube-api-access-qwsj7\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:32:43.311582 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.311178    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a20999e6-4f67-47b9-9cf1-da1b1fb027f2-serviceca\") pod \"node-ca-hnmxk\" (UID: \"a20999e6-4f67-47b9-9cf1-da1b1fb027f2\") " pod="openshift-image-registry/node-ca-hnmxk"
Oct 30 21:32:43.311582 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.311209    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-systemd-units\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.311582 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.311225    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7986120b-482f-4b0b-a605-32770569962d-ovnkube-config\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.311582 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.311238    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-sys\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.311582 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.311252    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-var-lib-kubelet\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.311582 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.311300    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjbfn\" (UniqueName: \"kubernetes.io/projected/032b9335-657e-4a86-aa6d-4ef46163014a-kube-api-access-xjbfn\") pod \"network-metrics-daemon-bxbq6\" (UID: \"032b9335-657e-4a86-aa6d-4ef46163014a\") " pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:32:43.311582 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.311328    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-hostroot\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.311582 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.311376    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-kubelet\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.311582 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.311396    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-host\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.311582 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.311442    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-sys-fs\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:32:43.311582 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.311465    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-system-cni-dir\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.311582 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.311480    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/752a05d6-f630-42ec-9795-0d28e9ed7d80-cnibin\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:32:43.312022 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.311495    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-etc-selinux\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:32:43.312022 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.311513    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6e5648ef-56ca-4d99-ad1a-0adb0fc50855-konnectivity-ca\") pod \"konnectivity-agent-h5xkl\" (UID: \"6e5648ef-56ca-4d99-ad1a-0adb0fc50855\") " pod="kube-system/konnectivity-agent-h5xkl"
Oct 30 21:32:43.312022 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.311540    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfn4f\" (UniqueName: \"kubernetes.io/projected/2a7ef1f3-4e9d-40db-93a3-4d38a3474e89-kube-api-access-kfn4f\") pod \"iptables-alerter-w9ckc\" (UID: \"2a7ef1f3-4e9d-40db-93a3-4d38a3474e89\") " pod="openshift-network-operator/iptables-alerter-w9ckc"
Oct 30 21:32:43.312022 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.311564    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-cni-netd\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.312022 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.311586    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-modprobe-d\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.312022 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.311606    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-systemd\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.312022 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.311620    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-cnibin\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.312022 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.311634    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-run-openvswitch\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.312022 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.311654    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-node-log\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.312022 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.311669    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7986120b-482f-4b0b-a605-32770569962d-ovn-node-metrics-cert\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.312022 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.311683    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7986120b-482f-4b0b-a605-32770569962d-ovnkube-script-lib\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.312022 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.311696    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6e5648ef-56ca-4d99-ad1a-0adb0fc50855-agent-certs\") pod \"konnectivity-agent-h5xkl\" (UID: \"6e5648ef-56ca-4d99-ad1a-0adb0fc50855\") " pod="kube-system/konnectivity-agent-h5xkl"
Oct 30 21:32:43.312022 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.311729    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-var-lib-cni-multus\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.334279 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.334260    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:32:43.334350 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.334291    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:32:43.334455 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.334439    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:32:43.334512 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.334496    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jnpcj\""
Oct 30 21:32:43.334698 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.334686    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6t698\""
Oct 30 21:32:43.334776 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.334764    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:32:43.334828 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.334773    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:32:43.334828 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.334784    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:32:43.334828 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.334792    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:32:43.334828 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.334782    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:32:43.334828 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.334823    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:32:43.334975 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.334827    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:32:43.338488 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.338473    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:32:43.338546 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.338476    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:32:43.338546 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.338476    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:32:43.338546 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.338528    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:32:43.338653 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.338561    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pplxq\""
Oct 30 21:32:43.338653 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.338614    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tvlkp\""
Oct 30 21:32:43.384781 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.384763    2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:32:43.384870 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.384799    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-j674l"
Oct 30 21:32:43.412015 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.411995    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-cnibin\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.412128 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412029    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-run-openvswitch\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.412128 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412055    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-node-log\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.412128 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412122    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-run-openvswitch\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.412258 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412124    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7986120b-482f-4b0b-a605-32770569962d-ovn-node-metrics-cert\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.412258 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412136    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-cnibin\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.412258 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412156    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-node-log\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.412258 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412197    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7986120b-482f-4b0b-a605-32770569962d-ovnkube-script-lib\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.412258 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412216    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6e5648ef-56ca-4d99-ad1a-0adb0fc50855-agent-certs\") pod \"konnectivity-agent-h5xkl\" (UID: \"6e5648ef-56ca-4d99-ad1a-0adb0fc50855\") " pod="kube-system/konnectivity-agent-h5xkl"
Oct 30 21:32:43.412258 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412254    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-var-lib-cni-multus\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.412604 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412341    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-var-lib-cni-multus\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.412604 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412389    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-run-multus-certs\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.412604 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412436    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkvjz\" (UniqueName: \"kubernetes.io/projected/9a1460f5-20f5-4bd1-843f-3838935f2ce2-kube-api-access-lkvjz\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.412604 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412455    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcz4j\" (UniqueName: \"kubernetes.io/projected/e1a448f1-e180-4fbf-9923-fac0b9dbec74-kube-api-access-xcz4j\") pod \"network-check-target-spx9v\" (UID: \"e1a448f1-e180-4fbf-9923-fac0b9dbec74\") " pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:32:43.412604 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412494    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-run-multus-certs\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.412604 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412529    2572 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:32:43.412885 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412641    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-run-systemd\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.412885 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412671    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs\") pod \"network-metrics-daemon-bxbq6\" (UID: \"032b9335-657e-4a86-aa6d-4ef46163014a\") " pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:32:43.412885 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412695    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:32:43.412885 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412721    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-slash\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.412885 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412724    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-run-systemd\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.412885 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412771    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-slash\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.412885 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:43.412804    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:32:43.412885 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412785    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:32:43.412885 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412828    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-original-pull-secret\") pod \"global-pull-secret-syncer-2z8sh\" (UID: \"2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49\") " pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:32:43.412885 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412833    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7986120b-482f-4b0b-a605-32770569962d-ovnkube-script-lib\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.412885 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:43.412866    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs podName:032b9335-657e-4a86-aa6d-4ef46163014a nodeName:}" failed. No retries permitted until 2026-10-30 21:32:43.912847011 +0000 UTC m=+2.295743568 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs") pod "network-metrics-daemon-bxbq6" (UID: "032b9335-657e-4a86-aa6d-4ef46163014a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:32:43.412885 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412884    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-socket-dir\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:32:43.413429 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412908    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-registration-dir\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:32:43.413429 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:43.412919    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:32:43.413429 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412929    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a1460f5-20f5-4bd1-843f-3838935f2ce2-cni-binary-copy\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.413429 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412949    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-multus-socket-dir-parent\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.413429 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412968    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-run-k8s-cni-cncf-io\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.413429 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412971    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-socket-dir\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:32:43.413429 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:43.412978    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-original-pull-secret podName:2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49 nodeName:}" failed. No retries permitted until 2026-10-30 21:32:43.912965297 +0000 UTC m=+2.295861853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-original-pull-secret") pod "global-pull-secret-syncer-2z8sh" (UID: "2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:32:43.413429 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412998    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-run-k8s-cni-cncf-io\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.413429 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.412997    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-registration-dir\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:32:43.413429 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413023    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a20999e6-4f67-47b9-9cf1-da1b1fb027f2-host\") pod \"node-ca-hnmxk\" (UID: \"a20999e6-4f67-47b9-9cf1-da1b1fb027f2\") " pod="openshift-image-registry/node-ca-hnmxk"
Oct 30 21:32:43.413429 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413041    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a20999e6-4f67-47b9-9cf1-da1b1fb027f2-host\") pod \"node-ca-hnmxk\" (UID: \"a20999e6-4f67-47b9-9cf1-da1b1fb027f2\") " pod="openshift-image-registry/node-ca-hnmxk"
Oct 30 21:32:43.413429 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413050    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-var-lib-openvswitch\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.413429 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413080    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-multus-socket-dir-parent\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.413429 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413074    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-etc-openvswitch\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.413429 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413119    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgwnq\" (UniqueName: \"kubernetes.io/projected/7986120b-482f-4b0b-a605-32770569962d-kube-api-access-pgwnq\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.413429 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413121    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-var-lib-openvswitch\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.413429 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413142    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-etc-openvswitch\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.414195 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413227    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-run\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.414195 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413263    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/752a05d6-f630-42ec-9795-0d28e9ed7d80-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:32:43.414195 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413286    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/061313cb-4d50-4ae5-afda-dc63a62a013b-tmp\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.414195 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413309    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rbpk\" (UniqueName: \"kubernetes.io/projected/23a040b8-2420-4d59-b426-f52832dc7de1-kube-api-access-7rbpk\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:32:43.414195 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413321    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-run\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.414195 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413330    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/239a77e8-635c-4388-ad79-1cb3b15129d8-tmp-dir\") pod \"node-resolver-9xqln\" (UID: \"239a77e8-635c-4388-ad79-1cb3b15129d8\") " pod="openshift-dns/node-resolver-9xqln"
Oct 30 21:32:43.414195 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413351    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-run-netns\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.414195 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413371    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-multus-conf-dir\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.414195 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413390    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-log-socket\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.414195 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413408    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-dbus\") pod \"global-pull-secret-syncer-2z8sh\" (UID: \"2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49\") " pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:32:43.414195 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413450    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/239a77e8-635c-4388-ad79-1cb3b15129d8-hosts-file\") pod \"node-resolver-9xqln\" (UID: \"239a77e8-635c-4388-ad79-1cb3b15129d8\") " pod="openshift-dns/node-resolver-9xqln"
Oct 30 21:32:43.414195 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413472    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t68x5\" (UniqueName: \"kubernetes.io/projected/239a77e8-635c-4388-ad79-1cb3b15129d8-kube-api-access-t68x5\") pod \"node-resolver-9xqln\" (UID: \"239a77e8-635c-4388-ad79-1cb3b15129d8\") " pod="openshift-dns/node-resolver-9xqln"
Oct 30 21:32:43.414195 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413493    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-sysctl-d\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.414195 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413518    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdkhn\" (UniqueName: \"kubernetes.io/projected/061313cb-4d50-4ae5-afda-dc63a62a013b-kube-api-access-qdkhn\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.414195 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413540    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-run-netns\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.414195 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413564    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-cni-bin\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.414195 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413602    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-run-netns\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.414195 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413653    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-multus-conf-dir\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.415035 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413668    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-sysctl-d\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.415035 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413642    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-sysconfig\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.415035 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413730    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/239a77e8-635c-4388-ad79-1cb3b15129d8-hosts-file\") pod \"node-resolver-9xqln\" (UID: \"239a77e8-635c-4388-ad79-1cb3b15129d8\") " pod="openshift-dns/node-resolver-9xqln"
Oct 30 21:32:43.415035 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413749    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-dbus\") pod \"global-pull-secret-syncer-2z8sh\" (UID: \"2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49\") " pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:32:43.415035 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413408    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a1460f5-20f5-4bd1-843f-3838935f2ce2-cni-binary-copy\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.415035 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413778    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-sysctl-conf\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.415035 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413801    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-device-dir\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:32:43.415035 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413808    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-cni-bin\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.415035 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413823    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/752a05d6-f630-42ec-9795-0d28e9ed7d80-cni-binary-copy\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:32:43.415035 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413829    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/239a77e8-635c-4388-ad79-1cb3b15129d8-tmp-dir\") pod \"node-resolver-9xqln\" (UID: \"239a77e8-635c-4388-ad79-1cb3b15129d8\") " pod="openshift-dns/node-resolver-9xqln"
Oct 30 21:32:43.415035 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413831    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/752a05d6-f630-42ec-9795-0d28e9ed7d80-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:32:43.415035 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413842    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-run-ovn\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.415035 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413882    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-run-netns\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.415035 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413892    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-run-ovn\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.415035 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413888    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-kubernetes\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.415035 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413697    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-log-socket\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.415035 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413926    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-kubernetes\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.415035 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413928    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-etc-kubernetes\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.415829 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413957    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/752a05d6-f630-42ec-9795-0d28e9ed7d80-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:32:43.415829 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413961    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-device-dir\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:32:43.415829 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413982    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a7ef1f3-4e9d-40db-93a3-4d38a3474e89-host-slash\") pod \"iptables-alerter-w9ckc\" (UID: \"2a7ef1f3-4e9d-40db-93a3-4d38a3474e89\") " pod="openshift-network-operator/iptables-alerter-w9ckc"
Oct 30 21:32:43.415829 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414001    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-sysconfig\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.415829 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414006    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-kubelet-config\") pod \"global-pull-secret-syncer-2z8sh\" (UID: \"2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49\") " pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:32:43.415829 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.413961    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-etc-kubernetes\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.415829 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414029    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-var-lib-kubelet\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.415829 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414042    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a7ef1f3-4e9d-40db-93a3-4d38a3474e89-host-slash\") pod \"iptables-alerter-w9ckc\" (UID: \"2a7ef1f3-4e9d-40db-93a3-4d38a3474e89\") " pod="openshift-network-operator/iptables-alerter-w9ckc"
Oct 30 21:32:43.415829 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414053    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2a7ef1f3-4e9d-40db-93a3-4d38a3474e89-iptables-alerter-script\") pod \"iptables-alerter-w9ckc\" (UID: \"2a7ef1f3-4e9d-40db-93a3-4d38a3474e89\") " pod="openshift-network-operator/iptables-alerter-w9ckc"
Oct 30 21:32:43.415829 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414068    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-sysctl-conf\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.415829 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414079    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-run-ovn-kubernetes\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.415829 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414088    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-kubelet-config\") pod \"global-pull-secret-syncer-2z8sh\" (UID: \"2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49\") " pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:32:43.415829 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414106    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.415829 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414108    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-var-lib-kubelet\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.415829 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414131    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-run-ovn-kubernetes\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.415829 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414171    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.415829 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414193    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-multus-cni-dir\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.416287 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414214    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-var-lib-cni-bin\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.416287 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414239    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/752a05d6-f630-42ec-9795-0d28e9ed7d80-system-cni-dir\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:32:43.416287 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414255    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-multus-cni-dir\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.416287 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414265    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/752a05d6-f630-42ec-9795-0d28e9ed7d80-tuning-conf-dir\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:32:43.416287 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414290    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkjl7\" (UniqueName: \"kubernetes.io/projected/a20999e6-4f67-47b9-9cf1-da1b1fb027f2-kube-api-access-lkjl7\") pod \"node-ca-hnmxk\" (UID: \"a20999e6-4f67-47b9-9cf1-da1b1fb027f2\") " pod="openshift-image-registry/node-ca-hnmxk"
Oct 30 21:32:43.416287 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414296    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/752a05d6-f630-42ec-9795-0d28e9ed7d80-system-cni-dir\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:32:43.416287 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414316    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7986120b-482f-4b0b-a605-32770569962d-env-overrides\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.416287 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414341    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-lib-modules\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.416287 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414361    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-tuned\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.416287 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414383    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9a1460f5-20f5-4bd1-843f-3838935f2ce2-multus-daemon-config\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.416287 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414384    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/752a05d6-f630-42ec-9795-0d28e9ed7d80-cni-binary-copy\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:32:43.416287 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414407    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwsj7\" (UniqueName: \"kubernetes.io/projected/752a05d6-f630-42ec-9795-0d28e9ed7d80-kube-api-access-qwsj7\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:32:43.416287 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414455    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/752a05d6-f630-42ec-9795-0d28e9ed7d80-tuning-conf-dir\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:32:43.416287 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414508    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-lib-modules\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.416287 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414315    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-var-lib-cni-bin\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.416287 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414542    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a20999e6-4f67-47b9-9cf1-da1b1fb027f2-serviceca\") pod \"node-ca-hnmxk\" (UID: \"a20999e6-4f67-47b9-9cf1-da1b1fb027f2\") " pod="openshift-image-registry/node-ca-hnmxk"
Oct 30 21:32:43.416287 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414568    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-systemd-units\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.416950 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414591    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7986120b-482f-4b0b-a605-32770569962d-ovnkube-config\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.416950 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414600    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2a7ef1f3-4e9d-40db-93a3-4d38a3474e89-iptables-alerter-script\") pod \"iptables-alerter-w9ckc\" (UID: \"2a7ef1f3-4e9d-40db-93a3-4d38a3474e89\") " pod="openshift-network-operator/iptables-alerter-w9ckc"
Oct 30 21:32:43.416950 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414613    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-sys\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.416950 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414682    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-systemd-units\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.416950 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414842    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7986120b-482f-4b0b-a605-32770569962d-env-overrides\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.416950 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.414904    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-sys\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.416950 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.415025    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-var-lib-kubelet\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.416950 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.415055    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjbfn\" (UniqueName: \"kubernetes.io/projected/032b9335-657e-4a86-aa6d-4ef46163014a-kube-api-access-xjbfn\") pod \"network-metrics-daemon-bxbq6\" (UID: \"032b9335-657e-4a86-aa6d-4ef46163014a\") " pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:32:43.416950 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.415080    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-hostroot\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.416950 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.415104    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-kubelet\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.416950 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.415109    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9a1460f5-20f5-4bd1-843f-3838935f2ce2-multus-daemon-config\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.416950 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.415128    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-host\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.416950 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.415151    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-sys-fs\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:32:43.416950 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.415157    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7986120b-482f-4b0b-a605-32770569962d-ovnkube-config\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.416950 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.415169    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-var-lib-kubelet\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.416950 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.415177    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-system-cni-dir\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.416950 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.415219    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-host\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.416950 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.415221    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-kubelet\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.417445 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.415229    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-sys-fs\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:32:43.417445 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.415080    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a20999e6-4f67-47b9-9cf1-da1b1fb027f2-serviceca\") pod \"node-ca-hnmxk\" (UID: \"a20999e6-4f67-47b9-9cf1-da1b1fb027f2\") " pod="openshift-image-registry/node-ca-hnmxk"
Oct 30 21:32:43.417445 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.415247    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/752a05d6-f630-42ec-9795-0d28e9ed7d80-cnibin\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:32:43.417445 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.415267    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-hostroot\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.417445 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.415274    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-etc-selinux\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:32:43.417445 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.415284    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-system-cni-dir\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.417445 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.415300    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6e5648ef-56ca-4d99-ad1a-0adb0fc50855-konnectivity-ca\") pod \"konnectivity-agent-h5xkl\" (UID: \"6e5648ef-56ca-4d99-ad1a-0adb0fc50855\") " pod="kube-system/konnectivity-agent-h5xkl"
Oct 30 21:32:43.417445 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.415306    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/752a05d6-f630-42ec-9795-0d28e9ed7d80-cnibin\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:32:43.417445 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.415335    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-etc-selinux\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:32:43.417445 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.415345    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfn4f\" (UniqueName: \"kubernetes.io/projected/2a7ef1f3-4e9d-40db-93a3-4d38a3474e89-kube-api-access-kfn4f\") pod \"iptables-alerter-w9ckc\" (UID: \"2a7ef1f3-4e9d-40db-93a3-4d38a3474e89\") " pod="openshift-network-operator/iptables-alerter-w9ckc"
Oct 30 21:32:43.417445 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.415379    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-cni-netd\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.417445 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.415403    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-modprobe-d\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.417445 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.415452    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-systemd\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.417445 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.415529    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-systemd\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.417445 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.415569    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-cni-netd\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.417445 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.415634    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-modprobe-d\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.417445 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.416589    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6e5648ef-56ca-4d99-ad1a-0adb0fc50855-konnectivity-ca\") pod \"konnectivity-agent-h5xkl\" (UID: \"6e5648ef-56ca-4d99-ad1a-0adb0fc50855\") " pod="kube-system/konnectivity-agent-h5xkl"
Oct 30 21:32:43.417445 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.416671    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7986120b-482f-4b0b-a605-32770569962d-ovn-node-metrics-cert\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.417904 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.416739    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-tuned\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.417904 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.416782    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/752a05d6-f630-42ec-9795-0d28e9ed7d80-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:32:43.417904 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.416795    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6e5648ef-56ca-4d99-ad1a-0adb0fc50855-agent-certs\") pod \"konnectivity-agent-h5xkl\" (UID: \"6e5648ef-56ca-4d99-ad1a-0adb0fc50855\") " pod="kube-system/konnectivity-agent-h5xkl"
Oct 30 21:32:43.417904 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.417521    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/061313cb-4d50-4ae5-afda-dc63a62a013b-tmp\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.469578 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:43.469525    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:32:43.469578 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:43.469544    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:32:43.469578 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:43.469553    2572 projected.go:196] Error preparing data for projected volume kube-api-access-xcz4j for pod openshift-network-diagnostics/network-check-target-spx9v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:32:43.469723 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:43.469607    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1a448f1-e180-4fbf-9923-fac0b9dbec74-kube-api-access-xcz4j podName:e1a448f1-e180-4fbf-9923-fac0b9dbec74 nodeName:}" failed. No retries permitted until 2026-10-30 21:32:43.96959303 +0000 UTC m=+2.352489614 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xcz4j" (UniqueName: "kubernetes.io/projected/e1a448f1-e180-4fbf-9923-fac0b9dbec74-kube-api-access-xcz4j") pod "network-check-target-spx9v" (UID: "e1a448f1-e180-4fbf-9923-fac0b9dbec74") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:32:43.472555 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.472175    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkvjz\" (UniqueName: \"kubernetes.io/projected/9a1460f5-20f5-4bd1-843f-3838935f2ce2-kube-api-access-lkvjz\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.473674 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.473404    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjbfn\" (UniqueName: \"kubernetes.io/projected/032b9335-657e-4a86-aa6d-4ef46163014a-kube-api-access-xjbfn\") pod \"network-metrics-daemon-bxbq6\" (UID: \"032b9335-657e-4a86-aa6d-4ef46163014a\") " pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:32:43.474817 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.474780    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t68x5\" (UniqueName: \"kubernetes.io/projected/239a77e8-635c-4388-ad79-1cb3b15129d8-kube-api-access-t68x5\") pod \"node-resolver-9xqln\" (UID: \"239a77e8-635c-4388-ad79-1cb3b15129d8\") " pod="openshift-dns/node-resolver-9xqln"
Oct 30 21:32:43.474817 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.474790    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfn4f\" (UniqueName: \"kubernetes.io/projected/2a7ef1f3-4e9d-40db-93a3-4d38a3474e89-kube-api-access-kfn4f\") pod \"iptables-alerter-w9ckc\" (UID: \"2a7ef1f3-4e9d-40db-93a3-4d38a3474e89\") " pod="openshift-network-operator/iptables-alerter-w9ckc"
Oct 30 21:32:43.475031 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.475012    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgwnq\" (UniqueName: \"kubernetes.io/projected/7986120b-482f-4b0b-a605-32770569962d-kube-api-access-pgwnq\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.475196 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.475175    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwsj7\" (UniqueName: \"kubernetes.io/projected/752a05d6-f630-42ec-9795-0d28e9ed7d80-kube-api-access-qwsj7\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:32:43.475933 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.475909    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rbpk\" (UniqueName: \"kubernetes.io/projected/23a040b8-2420-4d59-b426-f52832dc7de1-kube-api-access-7rbpk\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:32:43.476336 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.476315    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkjl7\" (UniqueName: \"kubernetes.io/projected/a20999e6-4f67-47b9-9cf1-da1b1fb027f2-kube-api-access-lkjl7\") pod \"node-ca-hnmxk\" (UID: \"a20999e6-4f67-47b9-9cf1-da1b1fb027f2\") " pod="openshift-image-registry/node-ca-hnmxk"
Oct 30 21:32:43.476532 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.476516    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdkhn\" (UniqueName: \"kubernetes.io/projected/061313cb-4d50-4ae5-afda-dc63a62a013b-kube-api-access-qdkhn\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.558526 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.558498    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h5xkl"
Oct 30 21:32:43.573678 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.573656    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-25lhq"
Oct 30 21:32:43.592062 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.592030    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:32:43.599799 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.599782    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9xqln"
Oct 30 21:32:43.608053 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.608037    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hnmxk"
Oct 30 21:32:43.624937 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.624922    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w9ckc"
Oct 30 21:32:43.636496 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:43.636472    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod239a77e8_635c_4388_ad79_1cb3b15129d8.slice/crio-84fe6a4a113567dd158164901cf840c5c2f1e97eff7e29deeaea43687d082b21 WatchSource:0}: Error finding container 84fe6a4a113567dd158164901cf840c5c2f1e97eff7e29deeaea43687d082b21: Status 404 returned error can't find the container with id 84fe6a4a113567dd158164901cf840c5c2f1e97eff7e29deeaea43687d082b21
Oct 30 21:32:43.637788 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:43.637758    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04840b9eb1acb1deca2b80fddf33a1d0.slice/crio-57a02ffca03a800228acf10ce6d2ebc4d848938007cfcf0eeeace4ae5152c715 WatchSource:0}: Error finding container 57a02ffca03a800228acf10ce6d2ebc4d848938007cfcf0eeeace4ae5152c715: Status 404 returned error can't find the container with id 57a02ffca03a800228acf10ce6d2ebc4d848938007cfcf0eeeace4ae5152c715
Oct 30 21:32:43.638060 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:43.638035    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c3091500d45dc621d6409761c7c97d.slice/crio-2759ef767a46a495622a063dbe42fd1c49810f1d394f62f7605596f4fe4529f8 WatchSource:0}: Error finding container 2759ef767a46a495622a063dbe42fd1c49810f1d394f62f7605596f4fe4529f8: Status 404 returned error can't find the container with id 2759ef767a46a495622a063dbe42fd1c49810f1d394f62f7605596f4fe4529f8
Oct 30 21:32:43.638498 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:43.638477    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda20999e6_4f67_47b9_9cf1_da1b1fb027f2.slice/crio-578d4b23aaf1d1759ba22d8d449973b2451e249370addb2320c94042f416b02e WatchSource:0}: Error finding container 578d4b23aaf1d1759ba22d8d449973b2451e249370addb2320c94042f416b02e: Status 404 returned error can't find the container with id 578d4b23aaf1d1759ba22d8d449973b2451e249370addb2320c94042f416b02e
Oct 30 21:32:43.641593 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:43.641569    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7ef1f3_4e9d_40db_93a3_4d38a3474e89.slice/crio-48b8e80d8364eab5f3806e15ec0306cbac919143c5cd2840df05403b1d771b95 WatchSource:0}: Error finding container 48b8e80d8364eab5f3806e15ec0306cbac919143c5cd2840df05403b1d771b95: Status 404 returned error can't find the container with id 48b8e80d8364eab5f3806e15ec0306cbac919143c5cd2840df05403b1d771b95
Oct 30 21:32:43.643171 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.643155    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:32:43.662683 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.662663    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:32:43.666177 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.666166    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:32:43.672546 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:43.672499    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7986120b_482f_4b0b_a605_32770569962d.slice/crio-aee7f142cb9248fa5e4c673b180f01bad80b209f93bccffc794f3473ec44cb1c WatchSource:0}: Error finding container aee7f142cb9248fa5e4c673b180f01bad80b209f93bccffc794f3473ec44cb1c: Status 404 returned error can't find the container with id aee7f142cb9248fa5e4c673b180f01bad80b209f93bccffc794f3473ec44cb1c
Oct 30 21:32:43.674436 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:43.674358    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061313cb_4d50_4ae5_afda_dc63a62a013b.slice/crio-d213823bc68cbc969e348468cc88e36481e3c7e6df32d5a4433c3a0c18d68b25 WatchSource:0}: Error finding container d213823bc68cbc969e348468cc88e36481e3c7e6df32d5a4433c3a0c18d68b25: Status 404 returned error can't find the container with id d213823bc68cbc969e348468cc88e36481e3c7e6df32d5a4433c3a0c18d68b25
Oct 30 21:32:43.674674 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:43.674645    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:32:43.683191 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:43.683170    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a040b8_2420_4d59_b426_f52832dc7de1.slice/crio-1deabd615573216006eaaa87f9f1473fe7c4c993b545315013612b8a44559d60 WatchSource:0}: Error finding container 1deabd615573216006eaaa87f9f1473fe7c4c993b545315013612b8a44559d60: Status 404 returned error can't find the container with id 1deabd615573216006eaaa87f9f1473fe7c4c993b545315013612b8a44559d60
Oct 30 21:32:44.975280 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:44.975243    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs\") pod \"network-metrics-daemon-bxbq6\" (UID: \"032b9335-657e-4a86-aa6d-4ef46163014a\") " pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:32:44.975280 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:44.975284    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-original-pull-secret\") pod \"global-pull-secret-syncer-2z8sh\" (UID: \"2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49\") " pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:32:44.975523 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:44.975410    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:32:44.975523 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:44.975420    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:32:44.975523 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:44.975468    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs podName:032b9335-657e-4a86-aa6d-4ef46163014a nodeName:}" failed. No retries permitted until 2026-10-30 21:32:45.975453061 +0000 UTC m=+3.302419278 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs") pod "network-metrics-daemon-bxbq6" (UID: "032b9335-657e-4a86-aa6d-4ef46163014a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:32:44.975523 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:44.975482    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-original-pull-secret podName:2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49 nodeName:}" failed. No retries permitted until 2026-10-30 21:32:45.975475262 +0000 UTC m=+3.302441473 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-original-pull-secret") pod "global-pull-secret-syncer-2z8sh" (UID: "2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:32:45.075743 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:45.075677    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcz4j\" (UniqueName: \"kubernetes.io/projected/e1a448f1-e180-4fbf-9923-fac0b9dbec74-kube-api-access-xcz4j\") pod \"network-check-target-spx9v\" (UID: \"e1a448f1-e180-4fbf-9923-fac0b9dbec74\") " pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:32:45.076271 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:45.075805    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:32:45.076271 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:45.075818    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:32:45.076271 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:45.075826    2572 projected.go:196] Error preparing data for projected volume kube-api-access-xcz4j for pod openshift-network-diagnostics/network-check-target-spx9v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:32:45.076271 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:45.075867    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1a448f1-e180-4fbf-9923-fac0b9dbec74-kube-api-access-xcz4j podName:e1a448f1-e180-4fbf-9923-fac0b9dbec74 nodeName:}" failed. No retries permitted until 2026-10-30 21:32:46.075856403 +0000 UTC m=+3.402822614 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xcz4j" (UniqueName: "kubernetes.io/projected/e1a448f1-e180-4fbf-9923-fac0b9dbec74-kube-api-access-xcz4j") pod "network-check-target-spx9v" (UID: "e1a448f1-e180-4fbf-9923-fac0b9dbec74") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:32:45.145606 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:45.145561    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjt99" event={"ID":"7986120b-482f-4b0b-a605-32770569962d","Type":"ContainerStarted","Data":"aee7f142cb9248fa5e4c673b180f01bad80b209f93bccffc794f3473ec44cb1c"}
Oct 30 21:32:45.146568 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:45.146550    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w9ckc" event={"ID":"2a7ef1f3-4e9d-40db-93a3-4d38a3474e89","Type":"ContainerStarted","Data":"48b8e80d8364eab5f3806e15ec0306cbac919143c5cd2840df05403b1d771b95"}
Oct 30 21:32:45.147479 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:45.147458    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal" event={"ID":"57c3091500d45dc621d6409761c7c97d","Type":"ContainerStarted","Data":"2759ef767a46a495622a063dbe42fd1c49810f1d394f62f7605596f4fe4529f8"}
Oct 30 21:32:45.148393 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:45.148369    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9xqln" event={"ID":"239a77e8-635c-4388-ad79-1cb3b15129d8","Type":"ContainerStarted","Data":"84fe6a4a113567dd158164901cf840c5c2f1e97eff7e29deeaea43687d082b21"}
Oct 30 21:32:45.149276 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:45.149258    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal" event={"ID":"04840b9eb1acb1deca2b80fddf33a1d0","Type":"ContainerStarted","Data":"57a02ffca03a800228acf10ce6d2ebc4d848938007cfcf0eeeace4ae5152c715"}
Oct 30 21:32:45.150284 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:45.150266    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hnmxk" event={"ID":"a20999e6-4f67-47b9-9cf1-da1b1fb027f2","Type":"ContainerStarted","Data":"578d4b23aaf1d1759ba22d8d449973b2451e249370addb2320c94042f416b02e"}
Oct 30 21:32:45.151231 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:45.151217    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j" event={"ID":"23a040b8-2420-4d59-b426-f52832dc7de1","Type":"ContainerStarted","Data":"1deabd615573216006eaaa87f9f1473fe7c4c993b545315013612b8a44559d60"}
Oct 30 21:32:45.152103 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:45.152088    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-86426" event={"ID":"061313cb-4d50-4ae5-afda-dc63a62a013b","Type":"ContainerStarted","Data":"d213823bc68cbc969e348468cc88e36481e3c7e6df32d5a4433c3a0c18d68b25"}
Oct 30 21:32:45.350711 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:45.350638    2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:32:45.443899 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:45.443852    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:27:44 +0000 UTC" deadline="2028-07-04 07:46:27.341702293 +0000 UTC"
Oct 30 21:32:45.443899 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:45.443891    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14698h13m41.89781339s"
Oct 30 21:32:45.461769 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:45.461734    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1460f5_20f5_4bd1_843f_3838935f2ce2.slice/crio-0ed851957e516501ee89a73862c5ac5353d106076614edd774fc99d801d9a022 WatchSource:0}: Error finding container 0ed851957e516501ee89a73862c5ac5353d106076614edd774fc99d801d9a022: Status 404 returned error can't find the container with id 0ed851957e516501ee89a73862c5ac5353d106076614edd774fc99d801d9a022
Oct 30 21:32:45.462380 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:45.462336    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e5648ef_56ca_4d99_ad1a_0adb0fc50855.slice/crio-875aebd2b801f95612c15ae050e4703436aafd5316b77c6c2ce7a315981a29bd WatchSource:0}: Error finding container 875aebd2b801f95612c15ae050e4703436aafd5316b77c6c2ce7a315981a29bd: Status 404 returned error can't find the container with id 875aebd2b801f95612c15ae050e4703436aafd5316b77c6c2ce7a315981a29bd
Oct 30 21:32:45.487645 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:32:45.487620    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752a05d6_f630_42ec_9795_0d28e9ed7d80.slice/crio-44128cde9864ff4f8f5f58931dcd7fa31cccfbe695c5adec39318d17a5bf4965 WatchSource:0}: Error finding container 44128cde9864ff4f8f5f58931dcd7fa31cccfbe695c5adec39318d17a5bf4965: Status 404 returned error can't find the container with id 44128cde9864ff4f8f5f58931dcd7fa31cccfbe695c5adec39318d17a5bf4965
Oct 30 21:32:45.625947 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:45.625874    2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:32:45.984031 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:45.983319    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs\") pod \"network-metrics-daemon-bxbq6\" (UID: \"032b9335-657e-4a86-aa6d-4ef46163014a\") " pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:32:45.984031 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:45.983407    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-original-pull-secret\") pod \"global-pull-secret-syncer-2z8sh\" (UID: \"2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49\") " pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:32:45.984031 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:45.983539    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:32:45.984031 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:45.983612    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs podName:032b9335-657e-4a86-aa6d-4ef46163014a nodeName:}" failed. No retries permitted until 2026-10-30 21:32:47.983592145 +0000 UTC m=+5.310558370 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs") pod "network-metrics-daemon-bxbq6" (UID: "032b9335-657e-4a86-aa6d-4ef46163014a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:32:45.984031 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:45.983540    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:32:45.984031 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:45.983658    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-original-pull-secret podName:2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49 nodeName:}" failed. No retries permitted until 2026-10-30 21:32:47.983647713 +0000 UTC m=+5.310613937 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-original-pull-secret") pod "global-pull-secret-syncer-2z8sh" (UID: "2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:32:46.083770 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:46.083725    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcz4j\" (UniqueName: \"kubernetes.io/projected/e1a448f1-e180-4fbf-9923-fac0b9dbec74-kube-api-access-xcz4j\") pod \"network-check-target-spx9v\" (UID: \"e1a448f1-e180-4fbf-9923-fac0b9dbec74\") " pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:32:46.084448 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:46.083939    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:32:46.084448 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:46.083958    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:32:46.084448 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:46.083970    2572 projected.go:196] Error preparing data for projected volume kube-api-access-xcz4j for pod openshift-network-diagnostics/network-check-target-spx9v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:32:46.084448 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:46.084036    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1a448f1-e180-4fbf-9923-fac0b9dbec74-kube-api-access-xcz4j podName:e1a448f1-e180-4fbf-9923-fac0b9dbec74 nodeName:}" failed. No retries permitted until 2026-10-30 21:32:48.084019403 +0000 UTC m=+5.410985616 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xcz4j" (UniqueName: "kubernetes.io/projected/e1a448f1-e180-4fbf-9923-fac0b9dbec74-kube-api-access-xcz4j") pod "network-check-target-spx9v" (UID: "e1a448f1-e180-4fbf-9923-fac0b9dbec74") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:32:46.142924 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:46.142890    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:32:46.143122 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:46.143050    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-spx9v" podUID="e1a448f1-e180-4fbf-9923-fac0b9dbec74"
Oct 30 21:32:46.143307 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:46.143140    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:32:46.143307 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:46.143214    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2z8sh" podUID="2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49"
Oct 30 21:32:46.143307 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:46.143273    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:32:46.143551 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:46.143379    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxbq6" podUID="032b9335-657e-4a86-aa6d-4ef46163014a"
Oct 30 21:32:46.177367 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:46.173487    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-956lw" event={"ID":"752a05d6-f630-42ec-9795-0d28e9ed7d80","Type":"ContainerStarted","Data":"44128cde9864ff4f8f5f58931dcd7fa31cccfbe695c5adec39318d17a5bf4965"}
Oct 30 21:32:46.186332 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:46.186301    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h5xkl" event={"ID":"6e5648ef-56ca-4d99-ad1a-0adb0fc50855","Type":"ContainerStarted","Data":"875aebd2b801f95612c15ae050e4703436aafd5316b77c6c2ce7a315981a29bd"}
Oct 30 21:32:46.193068 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:46.193039    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-25lhq" event={"ID":"9a1460f5-20f5-4bd1-843f-3838935f2ce2","Type":"ContainerStarted","Data":"0ed851957e516501ee89a73862c5ac5353d106076614edd774fc99d801d9a022"}
Oct 30 21:32:46.444857 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:46.444738    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:27:44 +0000 UTC" deadline="2028-08-05 05:03:13.130735704 +0000 UTC"
Oct 30 21:32:46.444857 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:46.444798    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15463h30m26.685942503s"
Oct 30 21:32:47.998565 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:47.997802    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs\") pod \"network-metrics-daemon-bxbq6\" (UID: \"032b9335-657e-4a86-aa6d-4ef46163014a\") " pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:32:47.998565 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:47.997852    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-original-pull-secret\") pod \"global-pull-secret-syncer-2z8sh\" (UID: \"2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49\") " pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:32:47.998565 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:47.997968    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:32:47.998565 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:47.998026    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-original-pull-secret podName:2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49 nodeName:}" failed. No retries permitted until 2026-10-30 21:32:51.998009394 +0000 UTC m=+9.324975617 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-original-pull-secret") pod "global-pull-secret-syncer-2z8sh" (UID: "2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:32:47.998565 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:47.998463    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:32:47.998565 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:47.998509    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs podName:032b9335-657e-4a86-aa6d-4ef46163014a nodeName:}" failed. No retries permitted until 2026-10-30 21:32:51.998494308 +0000 UTC m=+9.325460533 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs") pod "network-metrics-daemon-bxbq6" (UID: "032b9335-657e-4a86-aa6d-4ef46163014a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:32:48.099297 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:48.098865    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcz4j\" (UniqueName: \"kubernetes.io/projected/e1a448f1-e180-4fbf-9923-fac0b9dbec74-kube-api-access-xcz4j\") pod \"network-check-target-spx9v\" (UID: \"e1a448f1-e180-4fbf-9923-fac0b9dbec74\") " pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:32:48.099297 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:48.099109    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:32:48.099297 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:48.099130    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:32:48.099297 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:48.099165    2572 projected.go:196] Error preparing data for projected volume kube-api-access-xcz4j for pod openshift-network-diagnostics/network-check-target-spx9v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:32:48.099297 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:48.099220    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1a448f1-e180-4fbf-9923-fac0b9dbec74-kube-api-access-xcz4j podName:e1a448f1-e180-4fbf-9923-fac0b9dbec74 nodeName:}" failed. No retries permitted until 2026-10-30 21:32:52.099202735 +0000 UTC m=+9.426168960 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xcz4j" (UniqueName: "kubernetes.io/projected/e1a448f1-e180-4fbf-9923-fac0b9dbec74-kube-api-access-xcz4j") pod "network-check-target-spx9v" (UID: "e1a448f1-e180-4fbf-9923-fac0b9dbec74") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:32:48.142114 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:48.141688    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:32:48.142114 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:48.141823    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxbq6" podUID="032b9335-657e-4a86-aa6d-4ef46163014a"
Oct 30 21:32:48.142114 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:48.141893    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:32:48.142114 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:48.141958    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-spx9v" podUID="e1a448f1-e180-4fbf-9923-fac0b9dbec74"
Oct 30 21:32:48.142114 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:48.142011    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:32:48.142114 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:48.142079    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2z8sh" podUID="2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49"
Oct 30 21:32:50.142306 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:50.141914    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:32:50.142306 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:50.142062    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxbq6" podUID="032b9335-657e-4a86-aa6d-4ef46163014a"
Oct 30 21:32:50.142306 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:50.142135    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:32:50.142306 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:50.142187    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-spx9v" podUID="e1a448f1-e180-4fbf-9923-fac0b9dbec74"
Oct 30 21:32:50.142306 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:50.142228    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:32:50.142306 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:50.142276    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2z8sh" podUID="2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49"
Oct 30 21:32:50.219044 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:50.219004    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal" event={"ID":"57c3091500d45dc621d6409761c7c97d","Type":"ContainerStarted","Data":"3b2b55c7064b47a34cf4c24d5fbd4172bc700f8cd92a5c1ea9ed0b65222106a1"}
Oct 30 21:32:50.223649 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:50.223618    2572 generic.go:356] "Generic (PLEG): container finished" podID="04840b9eb1acb1deca2b80fddf33a1d0" containerID="ffe8166c062f1d8279eba9d50be23880cab39bb109adfbd71b590c4f386710e0" exitCode=0
Oct 30 21:32:50.223779 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:50.223700    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal" event={"ID":"04840b9eb1acb1deca2b80fddf33a1d0","Type":"ContainerDied","Data":"ffe8166c062f1d8279eba9d50be23880cab39bb109adfbd71b590c4f386710e0"}
Oct 30 21:32:50.226725 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:50.226698    2572 generic.go:356] "Generic (PLEG): container finished" podID="752a05d6-f630-42ec-9795-0d28e9ed7d80" containerID="ad5abbc3081b61b6528a8247fa6863218e1a950a079381330b84c53d98cf6240" exitCode=0
Oct 30 21:32:50.226840 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:50.226742    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-956lw" event={"ID":"752a05d6-f630-42ec-9795-0d28e9ed7d80","Type":"ContainerDied","Data":"ad5abbc3081b61b6528a8247fa6863218e1a950a079381330b84c53d98cf6240"}
Oct 30 21:32:50.453772 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:50.453677    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal" podStartSLOduration=7.453659074 podStartE2EDuration="7.453659074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:50.331545738 +0000 UTC m=+7.658511972" watchObservedRunningTime="2026-10-30 21:32:50.453659074 +0000 UTC m=+7.780625309"
Oct 30 21:32:51.237199 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:51.237164    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal" event={"ID":"04840b9eb1acb1deca2b80fddf33a1d0","Type":"ContainerStarted","Data":"108841fd68a8ca97ffb4121c526d3bd00f35b3a91ced5fbdb275db35fba7126f"}
Oct 30 21:32:51.240142 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:51.240115    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hnmxk" event={"ID":"a20999e6-4f67-47b9-9cf1-da1b1fb027f2","Type":"ContainerStarted","Data":"ea25325531a55d40fdf608503f8761bcc09c760c5e5909a4efec2d805b3251d8"}
Oct 30 21:32:51.242969 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:51.242944    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h5xkl" event={"ID":"6e5648ef-56ca-4d99-ad1a-0adb0fc50855","Type":"ContainerStarted","Data":"cbc7e82de0c5201558757396503f20baf2fea79058d502414288065ba7672347"}
Oct 30 21:32:51.468262 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:51.468167    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal" podStartSLOduration=8.468153271 podStartE2EDuration="8.468153271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:32:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:51.35494482 +0000 UTC m=+8.681911054" watchObservedRunningTime="2026-10-30 21:32:51.468153271 +0000 UTC m=+8.795119504"
Oct 30 21:32:51.588950 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:51.588845    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-h5xkl" podStartSLOduration=3.779955662 podStartE2EDuration="8.588823287s" totalImagesPullingTime="4.808867625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:51.468904024 +0000 UTC m=+8.795870257" watchObservedRunningTime="2026-10-30 21:32:51.588823287 +0000 UTC m=+8.915789521"
Oct 30 21:32:51.589329 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:51.589279    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hnmxk" podStartSLOduration=3.096555806 podStartE2EDuration="8.589269594s" totalImagesPullingTime="5.492713788s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:51.589260142 +0000 UTC m=+8.916226375" watchObservedRunningTime="2026-10-30 21:32:51.589269594 +0000 UTC m=+8.916235830"
Oct 30 21:32:52.030428 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:52.030365    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs\") pod \"network-metrics-daemon-bxbq6\" (UID: \"032b9335-657e-4a86-aa6d-4ef46163014a\") " pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:32:52.030428 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:52.030414    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-original-pull-secret\") pod \"global-pull-secret-syncer-2z8sh\" (UID: \"2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49\") " pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:32:52.030657 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:52.030552    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:32:52.030657 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:52.030609    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-original-pull-secret podName:2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:00.03059207 +0000 UTC m=+17.357558283 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-original-pull-secret") pod "global-pull-secret-syncer-2z8sh" (UID: "2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:32:52.031010 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:52.030989    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:32:52.031117 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:52.031046    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs podName:032b9335-657e-4a86-aa6d-4ef46163014a nodeName:}" failed. No retries permitted until 2026-10-30 21:33:00.031032542 +0000 UTC m=+17.357998755 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs") pod "network-metrics-daemon-bxbq6" (UID: "032b9335-657e-4a86-aa6d-4ef46163014a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:32:52.131602 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:52.131568    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcz4j\" (UniqueName: \"kubernetes.io/projected/e1a448f1-e180-4fbf-9923-fac0b9dbec74-kube-api-access-xcz4j\") pod \"network-check-target-spx9v\" (UID: \"e1a448f1-e180-4fbf-9923-fac0b9dbec74\") " pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:32:52.131775 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:52.131737    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:32:52.131775 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:52.131759    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:32:52.131775 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:52.131769    2572 projected.go:196] Error preparing data for projected volume kube-api-access-xcz4j for pod openshift-network-diagnostics/network-check-target-spx9v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:32:52.131915 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:52.131824    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1a448f1-e180-4fbf-9923-fac0b9dbec74-kube-api-access-xcz4j podName:e1a448f1-e180-4fbf-9923-fac0b9dbec74 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:00.131806342 +0000 UTC m=+17.458772573 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xcz4j" (UniqueName: "kubernetes.io/projected/e1a448f1-e180-4fbf-9923-fac0b9dbec74-kube-api-access-xcz4j") pod "network-check-target-spx9v" (UID: "e1a448f1-e180-4fbf-9923-fac0b9dbec74") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:32:52.142315 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:52.142281    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:32:52.142476 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:52.142417    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-spx9v" podUID="e1a448f1-e180-4fbf-9923-fac0b9dbec74"
Oct 30 21:32:52.142548 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:52.142476    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:32:52.142601 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:52.142551    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2z8sh" podUID="2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49"
Oct 30 21:32:52.142663 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:52.142602    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:32:52.142717 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:52.142681    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxbq6" podUID="032b9335-657e-4a86-aa6d-4ef46163014a"
Oct 30 21:32:52.246830 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:52.246793    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j" event={"ID":"23a040b8-2420-4d59-b426-f52832dc7de1","Type":"ContainerStarted","Data":"1fc4ef2c53aacdd5274f79b190373cf2ea2e4ebf7c14536d28ec30baf9e329e7"}
Oct 30 21:32:52.890637 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:52.890607    2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-h5xkl"
Oct 30 21:32:52.891050 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:52.891030    2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-h5xkl"
Oct 30 21:32:53.403935 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:53.403840    2572 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:32:54.138614 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:54.138466    2572 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:32:53.403864474Z","UUID":"22afc141-92d4-49b7-8aa0-c1951bd9e575","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:32:54.140062 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:54.140042    2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:32:54.140181 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:54.140070    2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:32:54.142723 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:54.142705    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:32:54.142822 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:54.142793    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2z8sh" podUID="2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49"
Oct 30 21:32:54.143132 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:54.143109    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:32:54.143220 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:54.143203    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxbq6" podUID="032b9335-657e-4a86-aa6d-4ef46163014a"
Oct 30 21:32:54.143259 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:54.143253    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:32:54.143308 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:54.143293    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-spx9v" podUID="e1a448f1-e180-4fbf-9923-fac0b9dbec74"
Oct 30 21:32:54.253094 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:54.253057    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9xqln" event={"ID":"239a77e8-635c-4388-ad79-1cb3b15129d8","Type":"ContainerStarted","Data":"550068c35b61af9518c7771fe367a9827e936c99490b7d0479582678b92ffe8d"}
Oct 30 21:32:54.255472 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:54.255424    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-25lhq" event={"ID":"9a1460f5-20f5-4bd1-843f-3838935f2ce2","Type":"ContainerStarted","Data":"67a11f2b53489a98a3d85f7e07bef88a9442f24da4eb5578b5988601ff2467d5"}
Oct 30 21:32:54.260220 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:54.260198    2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:32:54.260610 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:54.260581    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j" event={"ID":"23a040b8-2420-4d59-b426-f52832dc7de1","Type":"ContainerStarted","Data":"f38dbe78a524aa82ec19f57c350d2d08886587658669eae34fcddf14404a6d55"}
Oct 30 21:32:54.484129 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:54.483849    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-25lhq" podStartSLOduration=3.221418662 podStartE2EDuration="11.483828486s" totalImagesPullingTime="8.262409824s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:54.483608131 +0000 UTC m=+11.810574366" watchObservedRunningTime="2026-10-30 21:32:54.483828486 +0000 UTC m=+11.810794722"
Oct 30 21:32:54.484633 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:54.484423    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-9xqln" podStartSLOduration=2.998420689 podStartE2EDuration="11.484411993s" totalImagesPullingTime="8.485991304s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:54.365557687 +0000 UTC m=+11.692523920" watchObservedRunningTime="2026-10-30 21:32:54.484411993 +0000 UTC m=+11.811378227"
Oct 30 21:32:55.265016 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:55.264942    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j" event={"ID":"23a040b8-2420-4d59-b426-f52832dc7de1","Type":"ContainerStarted","Data":"6ad59ff07446bd812009ec7d55c8406a1028b107c5e2c85334bd429b6f10d83a"}
Oct 30 21:32:55.267173 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:55.267146    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w9ckc" event={"ID":"2a7ef1f3-4e9d-40db-93a3-4d38a3474e89","Type":"ContainerStarted","Data":"d45dff639725afea5f7e3b065edebf3790e0c691c3763621bf7f41338a152643"}
Oct 30 21:32:55.269024 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:55.268997    2572 generic.go:356] "Generic (PLEG): container finished" podID="752a05d6-f630-42ec-9795-0d28e9ed7d80" containerID="33c75b63ac9e8b43036f5be84f365f19874f7b06165f267279e372fdcc2e4cd6" exitCode=0
Oct 30 21:32:55.269111 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:55.269073    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-956lw" event={"ID":"752a05d6-f630-42ec-9795-0d28e9ed7d80","Type":"ContainerDied","Data":"33c75b63ac9e8b43036f5be84f365f19874f7b06165f267279e372fdcc2e4cd6"}
Oct 30 21:32:55.380084 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:55.380018    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j" podStartSLOduration=2.570770325 podStartE2EDuration="12.379999379s" totalImagesPullingTime="9.809229054s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:32:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:55.379963699 +0000 UTC m=+12.706929932" watchObservedRunningTime="2026-10-30 21:32:55.379999379 +0000 UTC m=+12.706965614"
Oct 30 21:32:55.610552 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:55.610511    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-w9ckc" podStartSLOduration=3.807833914 podStartE2EDuration="12.610497952s" totalImagesPullingTime="8.802664038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:55.610258703 +0000 UTC m=+12.937224935" watchObservedRunningTime="2026-10-30 21:32:55.610497952 +0000 UTC m=+12.937464185"
Oct 30 21:32:56.142697 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:56.142663    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:32:56.142860 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:56.142770    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:32:56.142860 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:56.142785    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:32:56.142990 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:56.142873    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2z8sh" podUID="2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49"
Oct 30 21:32:56.142990 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:56.142970    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-spx9v" podUID="e1a448f1-e180-4fbf-9923-fac0b9dbec74"
Oct 30 21:32:56.143091 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:56.143059    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxbq6" podUID="032b9335-657e-4a86-aa6d-4ef46163014a"
Oct 30 21:32:56.274029 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:56.273987    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-86426" event={"ID":"061313cb-4d50-4ae5-afda-dc63a62a013b","Type":"ContainerStarted","Data":"2a0cb244ab825722adce259ed5d2c37427fb3a1b443e0051795a859a79e317e3"}
Oct 30 21:32:56.400080 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:56.399986    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-86426" podStartSLOduration=2.5945127 podStartE2EDuration="13.399970376s" totalImagesPullingTime="10.805457676s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:32:56.399423805 +0000 UTC m=+13.726390040" watchObservedRunningTime="2026-10-30 21:32:56.399970376 +0000 UTC m=+13.726936608"
Oct 30 21:32:57.278552 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:57.278510    2572 generic.go:356] "Generic (PLEG): container finished" podID="752a05d6-f630-42ec-9795-0d28e9ed7d80" containerID="5b0fc9b4dfae26dd49b535ae3d39287c0636d60510c6eac1c7c3e1b176f5320a" exitCode=0
Oct 30 21:32:57.279167 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:57.278570    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-956lw" event={"ID":"752a05d6-f630-42ec-9795-0d28e9ed7d80","Type":"ContainerDied","Data":"5b0fc9b4dfae26dd49b535ae3d39287c0636d60510c6eac1c7c3e1b176f5320a"}
Oct 30 21:32:58.142753 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:58.142728    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:32:58.142931 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:58.142773    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:32:58.142931 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:58.142788    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:32:58.142931 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:58.142919    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxbq6" podUID="032b9335-657e-4a86-aa6d-4ef46163014a"
Oct 30 21:32:58.143085 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:58.143034    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2z8sh" podUID="2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49"
Oct 30 21:32:58.143132 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:32:58.143111    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-spx9v" podUID="e1a448f1-e180-4fbf-9923-fac0b9dbec74"
Oct 30 21:32:59.286316 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:59.286063    2572 generic.go:356] "Generic (PLEG): container finished" podID="752a05d6-f630-42ec-9795-0d28e9ed7d80" containerID="31916e0f83c7a5cc25d29dff3528113f22c5ac455bc8260b9c0ed0cc2a572207" exitCode=0
Oct 30 21:32:59.286316 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:32:59.286151    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-956lw" event={"ID":"752a05d6-f630-42ec-9795-0d28e9ed7d80","Type":"ContainerDied","Data":"31916e0f83c7a5cc25d29dff3528113f22c5ac455bc8260b9c0ed0cc2a572207"}
Oct 30 21:33:00.097513 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:00.097479    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs\") pod \"network-metrics-daemon-bxbq6\" (UID: \"032b9335-657e-4a86-aa6d-4ef46163014a\") " pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:33:00.097513 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:00.097512    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-original-pull-secret\") pod \"global-pull-secret-syncer-2z8sh\" (UID: \"2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49\") " pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:33:00.097699 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:00.097615    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:33:00.097699 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:00.097614    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:33:00.097699 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:00.097669    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs podName:032b9335-657e-4a86-aa6d-4ef46163014a nodeName:}" failed. No retries permitted until 2026-10-30 21:33:16.097654862 +0000 UTC m=+33.424621072 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs") pod "network-metrics-daemon-bxbq6" (UID: "032b9335-657e-4a86-aa6d-4ef46163014a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:33:00.097699 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:00.097682    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-original-pull-secret podName:2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:16.097676451 +0000 UTC m=+33.424642662 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-original-pull-secret") pod "global-pull-secret-syncer-2z8sh" (UID: "2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:33:00.142622 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:00.142598    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:33:00.142713 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:00.142599    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:33:00.142754 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:00.142716    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxbq6" podUID="032b9335-657e-4a86-aa6d-4ef46163014a"
Oct 30 21:33:00.142754 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:00.142725    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:33:00.142816 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:00.142790    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-spx9v" podUID="e1a448f1-e180-4fbf-9923-fac0b9dbec74"
Oct 30 21:33:00.142879 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:00.142860    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2z8sh" podUID="2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49"
Oct 30 21:33:00.197787 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:00.197751    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcz4j\" (UniqueName: \"kubernetes.io/projected/e1a448f1-e180-4fbf-9923-fac0b9dbec74-kube-api-access-xcz4j\") pod \"network-check-target-spx9v\" (UID: \"e1a448f1-e180-4fbf-9923-fac0b9dbec74\") " pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:33:00.197928 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:00.197915    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:33:00.198018 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:00.197939    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:33:00.198018 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:00.197949    2572 projected.go:196] Error preparing data for projected volume kube-api-access-xcz4j for pod openshift-network-diagnostics/network-check-target-spx9v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:33:00.198018 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:00.197993    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e1a448f1-e180-4fbf-9923-fac0b9dbec74-kube-api-access-xcz4j podName:e1a448f1-e180-4fbf-9923-fac0b9dbec74 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:16.197980678 +0000 UTC m=+33.524946889 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xcz4j" (UniqueName: "kubernetes.io/projected/e1a448f1-e180-4fbf-9923-fac0b9dbec74-kube-api-access-xcz4j") pod "network-check-target-spx9v" (UID: "e1a448f1-e180-4fbf-9923-fac0b9dbec74") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:33:01.293329 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:01.293106    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 21:33:01.293788 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:01.293640    2572 generic.go:356] "Generic (PLEG): container finished" podID="7986120b-482f-4b0b-a605-32770569962d" containerID="7fa32e4295690d9ce92762231369af00c9e3bd979b045ddeb844417b0dc5c82c" exitCode=1
Oct 30 21:33:01.293788 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:01.293672    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjt99" event={"ID":"7986120b-482f-4b0b-a605-32770569962d","Type":"ContainerStarted","Data":"3abe12d8688e768138d0c2e31fb65f2d26c38ffb1a7564fc64ccdb257dd8bcbc"}
Oct 30 21:33:01.293788 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:01.293731    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjt99" event={"ID":"7986120b-482f-4b0b-a605-32770569962d","Type":"ContainerStarted","Data":"ef71c3bf8d7b31118084a77264038244700f190c92a680d8e832d72fdcedb3b5"}
Oct 30 21:33:01.293788 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:01.293773    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjt99" event={"ID":"7986120b-482f-4b0b-a605-32770569962d","Type":"ContainerStarted","Data":"86a148877e7958dd50117ec3f0b2e33bf2d63cf93e61780d79e8f22f0c357d16"}
Oct 30 21:33:01.293986 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:01.293799    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjt99" event={"ID":"7986120b-482f-4b0b-a605-32770569962d","Type":"ContainerStarted","Data":"ca2b876bd2ccf0c27b2f796a5d0f42d5a1aa818e80f4f6b1e4cdd7c837b0530b"}
Oct 30 21:33:01.293986 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:01.293827    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjt99" event={"ID":"7986120b-482f-4b0b-a605-32770569962d","Type":"ContainerDied","Data":"7fa32e4295690d9ce92762231369af00c9e3bd979b045ddeb844417b0dc5c82c"}
Oct 30 21:33:01.293986 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:01.293862    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjt99" event={"ID":"7986120b-482f-4b0b-a605-32770569962d","Type":"ContainerStarted","Data":"3e7d95b2c65433fea62ec1d44651edf7f2858a6be881e13d5531bd89ef32f331"}
Oct 30 21:33:02.141938 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:02.141909    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:33:02.141938 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:02.141937    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:33:02.142170 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:02.142004    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:33:02.142170 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:02.142019    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-spx9v" podUID="e1a448f1-e180-4fbf-9923-fac0b9dbec74"
Oct 30 21:33:02.142170 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:02.142094    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2z8sh" podUID="2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49"
Oct 30 21:33:02.142170 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:02.142141    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxbq6" podUID="032b9335-657e-4a86-aa6d-4ef46163014a"
Oct 30 21:33:02.299171 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:02.299142    2572 generic.go:356] "Generic (PLEG): container finished" podID="752a05d6-f630-42ec-9795-0d28e9ed7d80" containerID="865fab4defdcaa4ee2132c939d8efc22b43d74ab7da4f6812aa24c9beac08cf1" exitCode=0
Oct 30 21:33:02.299626 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:02.299189    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-956lw" event={"ID":"752a05d6-f630-42ec-9795-0d28e9ed7d80","Type":"ContainerDied","Data":"865fab4defdcaa4ee2132c939d8efc22b43d74ab7da4f6812aa24c9beac08cf1"}
Oct 30 21:33:03.304656 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:03.304620    2572 generic.go:356] "Generic (PLEG): container finished" podID="752a05d6-f630-42ec-9795-0d28e9ed7d80" containerID="045d16bc1eb4a855c2057cdeed55436899647a11ebc902b6bbabde8f04dc4eb7" exitCode=0
Oct 30 21:33:03.305129 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:03.304705    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-956lw" event={"ID":"752a05d6-f630-42ec-9795-0d28e9ed7d80","Type":"ContainerDied","Data":"045d16bc1eb4a855c2057cdeed55436899647a11ebc902b6bbabde8f04dc4eb7"}
Oct 30 21:33:03.307901 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:03.307884    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 21:33:03.308233 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:03.308205    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjt99" event={"ID":"7986120b-482f-4b0b-a605-32770569962d","Type":"ContainerStarted","Data":"c563149e0efd421683de2ac841eae9552ff900a6d9db305714f8e3f01d24e9ec"}
Oct 30 21:33:04.142136 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:04.142061    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:33:04.142136 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:04.142128    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:33:04.142363 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:04.142227    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:33:04.142363 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:04.142236    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxbq6" podUID="032b9335-657e-4a86-aa6d-4ef46163014a"
Oct 30 21:33:04.142363 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:04.142312    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-spx9v" podUID="e1a448f1-e180-4fbf-9923-fac0b9dbec74"
Oct 30 21:33:04.142479 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:04.142364    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2z8sh" podUID="2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49"
Oct 30 21:33:04.313647 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:04.313609    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-956lw" event={"ID":"752a05d6-f630-42ec-9795-0d28e9ed7d80","Type":"ContainerStarted","Data":"4d4ffd22c24d2099d8102a4ccdd99bbcff9eaffa91f18efbc7d452b8819388b7"}
Oct 30 21:33:04.433140 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:04.433051    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-956lw" podStartSLOduration=9.223619373 podStartE2EDuration="21.433039453s" totalImagesPullingTime="12.20942008s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:32:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:04.432489938 +0000 UTC m=+21.759456172" watchObservedRunningTime="2026-10-30 21:33:04.433039453 +0000 UTC m=+21.760005685"
Oct 30 21:33:05.320519 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:05.320491    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 21:33:05.321236 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:05.320963    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjt99" event={"ID":"7986120b-482f-4b0b-a605-32770569962d","Type":"ContainerStarted","Data":"f89428e9e2162211ddcea20634b0f861e36cb31cca027af1ce70502ff1f20fbd"}
Oct 30 21:33:05.321628 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:05.321607    2572 scope.go:122] "RemoveContainer" containerID="7fa32e4295690d9ce92762231369af00c9e3bd979b045ddeb844417b0dc5c82c"
Oct 30 21:33:06.141941 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:06.141755    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:33:06.142176 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:06.141769    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:33:06.142176 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:06.142025    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxbq6" podUID="032b9335-657e-4a86-aa6d-4ef46163014a"
Oct 30 21:33:06.142176 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:06.141815    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:33:06.142176 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:06.142099    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-spx9v" podUID="e1a448f1-e180-4fbf-9923-fac0b9dbec74"
Oct 30 21:33:06.142176 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:06.142166    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2z8sh" podUID="2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49"
Oct 30 21:33:06.326200 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:06.326170    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 21:33:06.326611 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:06.326556    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bjt99" event={"ID":"7986120b-482f-4b0b-a605-32770569962d","Type":"ContainerStarted","Data":"d5807b7d7f6b82208f41514a8c3038ba749131bcca1e4c9ae2d0ac699b2ee1e6"}
Oct 30 21:33:06.326875 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:06.326854    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:06.326991 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:06.326931    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:06.326991 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:06.326979    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:06.342232 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:06.342207    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:06.342388 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:06.342308    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:06.535554 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:06.535442    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bjt99" podStartSLOduration=7.775257881 podStartE2EDuration="23.535418079s" totalImagesPullingTime="15.760160198s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:06.534234618 +0000 UTC m=+23.861200862" watchObservedRunningTime="2026-10-30 21:33:06.535418079 +0000 UTC m=+23.862384313"
Oct 30 21:33:07.833485 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:07.833451    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2z8sh"]
Oct 30 21:33:07.833885 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:07.833596    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:33:07.833885 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:07.833692    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2z8sh" podUID="2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49"
Oct 30 21:33:07.836553 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:07.836529    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-spx9v"]
Oct 30 21:33:07.836679 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:07.836629    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:33:07.836740 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:07.836719    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-spx9v" podUID="e1a448f1-e180-4fbf-9923-fac0b9dbec74"
Oct 30 21:33:07.882873 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:07.882853    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bxbq6"]
Oct 30 21:33:07.882974 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:07.882939    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:33:07.883021 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:07.883006    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxbq6" podUID="032b9335-657e-4a86-aa6d-4ef46163014a"
Oct 30 21:33:09.142540 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:09.142509    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:33:09.142910 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:09.142606    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-spx9v" podUID="e1a448f1-e180-4fbf-9923-fac0b9dbec74"
Oct 30 21:33:10.141727 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:10.141657    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:33:10.141855 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:10.141766    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2z8sh" podUID="2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49"
Oct 30 21:33:10.141855 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:10.141820    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:33:10.141930 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:10.141913    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxbq6" podUID="032b9335-657e-4a86-aa6d-4ef46163014a"
Oct 30 21:33:11.142031 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:11.142005    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:33:11.142466 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:11.142099    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-spx9v" podUID="e1a448f1-e180-4fbf-9923-fac0b9dbec74"
Oct 30 21:33:12.142553 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:12.142520    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:33:12.142939 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:12.142534    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:33:12.142939 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:12.142643    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxbq6" podUID="032b9335-657e-4a86-aa6d-4ef46163014a"
Oct 30 21:33:12.142939 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:12.142684    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-2z8sh" podUID="2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49"
Oct 30 21:33:13.142780 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.142743    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:33:13.143250 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:13.142843    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-spx9v" podUID="e1a448f1-e180-4fbf-9923-fac0b9dbec74"
Oct 30 21:33:13.487008 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.486982    2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeReady"
Oct 30 21:33:13.487981 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.487711    2572 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 21:33:13.645199 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.645162    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9trcs"]
Oct 30 21:33:13.706951 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.706883    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9m2lz"]
Oct 30 21:33:13.707096 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.707026    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-9trcs"
Oct 30 21:33:13.720960 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.720938    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cx7d4"]
Oct 30 21:33:13.721079 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.721051    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-9m2lz"
Oct 30 21:33:13.739672 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.739650    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-54cbfc65d5-sg48h"]
Oct 30 21:33:13.739775 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.739752    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-cx7d4"
Oct 30 21:33:13.759705 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.759682    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48"]
Oct 30 21:33:13.759860 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.759793    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:13.764502 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.763172    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 21:33:13.766269 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.766246    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 21:33:13.781407 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.781382    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-qx7zq"]
Oct 30 21:33:13.781522 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.781500    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48"
Oct 30 21:33:13.793965 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.793947    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-z4jbc"]
Oct 30 21:33:13.794063 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.794049    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-qx7zq"
Oct 30 21:33:13.799626 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.799605    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzjqd\" (UniqueName: \"kubernetes.io/projected/364111d6-7289-47be-8cf0-c10c7b6fbcdc-kube-api-access-lzjqd\") pod \"kas-connection-checker-685895c7c9-9trcs\" (UID: \"364111d6-7289-47be-8cf0-c10c7b6fbcdc\") " pod="kube-system/kas-connection-checker-685895c7c9-9trcs"
Oct 30 21:33:13.799716 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.799668    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28w48\" (UniqueName: \"kubernetes.io/projected/8a74b249-57da-4405-af8e-ae777a44415e-kube-api-access-28w48\") pod \"kas-connection-checker-685895c7c9-9m2lz\" (UID: \"8a74b249-57da-4405-af8e-ae777a44415e\") " pod="kube-system/kas-connection-checker-685895c7c9-9m2lz"
Oct 30 21:33:13.808222 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.808208    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-bn8fx"]
Oct 30 21:33:13.808328 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.808313    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:13.815407 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.815391    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 21:33:13.815609 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.815598    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 21:33:13.815779 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.815767    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 21:33:13.815864 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.815849    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8l7jw\""
Oct 30 21:33:13.822374 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.822335    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-z8clf"]
Oct 30 21:33:13.822483 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.822427    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bn8fx"
Oct 30 21:33:13.836819 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.836799    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd"]
Oct 30 21:33:13.836923 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.836911    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf"
Oct 30 21:33:13.854490 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.854470    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fgpcv"]
Oct 30 21:33:13.854604 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.854588    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd"
Oct 30 21:33:13.863041 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.863021    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 21:33:13.863236 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.863218    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 21:33:13.863236 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.863232    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 21:33:13.863367 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.863228    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:33:13.863367 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.863260    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 21:33:13.863367 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.863269    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 21:33:13.863367 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.863250    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:33:13.863541 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.863528    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 21:33:13.863616 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.863601    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 21:33:13.865376 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.865364    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 21:33:13.875731 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.875716    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc"]
Oct 30 21:33:13.875848 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.875834    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fgpcv"
Oct 30 21:33:13.889867 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.889845    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-74b44b76d6-l7p76"]
Oct 30 21:33:13.889950 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.889938    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc"
Oct 30 21:33:13.900928 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.900912    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f54f9754-2024-41b9-8f84-fc3d02a22253-service-ca-bundle\") pod \"insights-operator-7c4986d449-z4jbc\" (UID: \"f54f9754-2024-41b9-8f84-fc3d02a22253\") " pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:13.901014 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.900937    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm65m\" (UniqueName: \"kubernetes.io/projected/f54f9754-2024-41b9-8f84-fc3d02a22253-kube-api-access-tm65m\") pod \"insights-operator-7c4986d449-z4jbc\" (UID: \"f54f9754-2024-41b9-8f84-fc3d02a22253\") " pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:13.901014 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.900958    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbqkr\" (UniqueName: \"kubernetes.io/projected/f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef-kube-api-access-fbqkr\") pod \"service-ca-operator-544d749c8c-bn8fx\" (UID: \"f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bn8fx"
Oct 30 21:33:13.901083 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.901019    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f54f9754-2024-41b9-8f84-fc3d02a22253-snapshots\") pod \"insights-operator-7c4986d449-z4jbc\" (UID: \"f54f9754-2024-41b9-8f84-fc3d02a22253\") " pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:13.901083 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.901058    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f54f9754-2024-41b9-8f84-fc3d02a22253-serving-cert\") pod \"insights-operator-7c4986d449-z4jbc\" (UID: \"f54f9754-2024-41b9-8f84-fc3d02a22253\") " pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:13.901141 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.901088    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57sgv\" (UniqueName: \"kubernetes.io/projected/f84984eb-c144-4394-bb77-afc0c94b3a3a-kube-api-access-57sgv\") pod \"kas-connection-checker-685895c7c9-cx7d4\" (UID: \"f84984eb-c144-4394-bb77-afc0c94b3a3a\") " pod="kube-system/kas-connection-checker-685895c7c9-cx7d4"
Oct 30 21:33:13.901141 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.901120    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f54f9754-2024-41b9-8f84-fc3d02a22253-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-z4jbc\" (UID: \"f54f9754-2024-41b9-8f84-fc3d02a22253\") " pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:13.901197 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.901143    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-default-certificate\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:13.901197 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.901168    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wd6v\" (UniqueName: \"kubernetes.io/projected/83917e8e-50b6-4c52-b337-6f1be5a2c26a-kube-api-access-9wd6v\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:13.901256 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.901223    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d5f25065-34c9-4724-8dff-9ace0890d565-nginx-conf\") pod \"networking-console-plugin-86c478bb88-qx7zq\" (UID: \"d5f25065-34c9-4724-8dff-9ace0890d565\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qx7zq"
Oct 30 21:33:13.901256 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.901246    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28w48\" (UniqueName: \"kubernetes.io/projected/8a74b249-57da-4405-af8e-ae777a44415e-kube-api-access-28w48\") pod \"kas-connection-checker-685895c7c9-9m2lz\" (UID: \"8a74b249-57da-4405-af8e-ae777a44415e\") " pod="kube-system/kas-connection-checker-685895c7c9-9m2lz"
Oct 30 21:33:13.901365 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.901266    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-stats-auth\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:13.901365 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.901327    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2qz48\" (UID: \"9f16bd1f-a3bd-45ee-a02e-cb6b9e982327\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48"
Oct 30 21:33:13.901429 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.901368    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d5f25065-34c9-4724-8dff-9ace0890d565-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qx7zq\" (UID: \"d5f25065-34c9-4724-8dff-9ace0890d565\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qx7zq"
Oct 30 21:33:13.901429 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.901404    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzjqd\" (UniqueName: \"kubernetes.io/projected/364111d6-7289-47be-8cf0-c10c7b6fbcdc-kube-api-access-lzjqd\") pod \"kas-connection-checker-685895c7c9-9trcs\" (UID: \"364111d6-7289-47be-8cf0-c10c7b6fbcdc\") " pod="kube-system/kas-connection-checker-685895c7c9-9trcs"
Oct 30 21:33:13.901497 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.901429    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83917e8e-50b6-4c52-b337-6f1be5a2c26a-service-ca-bundle\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:13.901497 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.901463    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef-config\") pod \"service-ca-operator-544d749c8c-bn8fx\" (UID: \"f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bn8fx"
Oct 30 21:33:13.901580 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.901566    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef-serving-cert\") pod \"service-ca-operator-544d749c8c-bn8fx\" (UID: \"f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bn8fx"
Oct 30 21:33:13.901613 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.901604    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2qz48\" (UID: \"9f16bd1f-a3bd-45ee-a02e-cb6b9e982327\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48"
Oct 30 21:33:13.901654 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.901644    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f54f9754-2024-41b9-8f84-fc3d02a22253-tmp\") pod \"insights-operator-7c4986d449-z4jbc\" (UID: \"f54f9754-2024-41b9-8f84-fc3d02a22253\") " pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:13.901706 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.901693    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-metrics-certs\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:13.901736 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.901715    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk9wl\" (UniqueName: \"kubernetes.io/projected/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-kube-api-access-hk9wl\") pod \"cluster-monitoring-operator-884576c57-2qz48\" (UID: \"9f16bd1f-a3bd-45ee-a02e-cb6b9e982327\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48"
Oct 30 21:33:13.902770 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.902756    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7fc9466475-jz9vh"]
Oct 30 21:33:13.902862 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.902851    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:13.913150 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.913132    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 21:33:13.913236 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.913140    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:33:13.913236 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.913170    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 21:33:13.913236 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.913189    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 21:33:13.915875 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.915856    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 21:33:13.915937 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.915859    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 21:33:13.916130 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.916117    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-f9hfm\""
Oct 30 21:33:13.916130 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.916118    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 21:33:13.916230 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.916153    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 21:33:13.916230 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.916188    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5vdw5\""
Oct 30 21:33:13.916230 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.916221    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 21:33:13.916332 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.916234    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 21:33:13.916332 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.916239    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-lbvv2\""
Oct 30 21:33:13.916332 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.916243    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-hlsb4\""
Oct 30 21:33:13.916332 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.916253    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-69h9r\""
Oct 30 21:33:13.916332 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.916308    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-vxknp\""
Oct 30 21:33:13.926709 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.926691    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-64b9m"]
Oct 30 21:33:13.926819 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.926805    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:13.927734 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.927716    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-hlnd6\""
Oct 30 21:33:13.942105 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.942079    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 21:33:13.942516 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.942491    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-bj8tb\""
Oct 30 21:33:13.949547 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.949526    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5kfvz"]
Oct 30 21:33:13.949661 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.949646    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-64b9m"
Oct 30 21:33:13.954756 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.954730    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 21:33:13.954756 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.954754    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 21:33:13.954971 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.954955    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 21:33:13.954971 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.954969    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9vgdc\""
Oct 30 21:33:13.959135 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.959087    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzjqd\" (UniqueName: \"kubernetes.io/projected/364111d6-7289-47be-8cf0-c10c7b6fbcdc-kube-api-access-lzjqd\") pod \"kas-connection-checker-685895c7c9-9trcs\" (UID: \"364111d6-7289-47be-8cf0-c10c7b6fbcdc\") " pod="kube-system/kas-connection-checker-685895c7c9-9trcs"
Oct 30 21:33:13.959135 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.959090    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28w48\" (UniqueName: \"kubernetes.io/projected/8a74b249-57da-4405-af8e-ae777a44415e-kube-api-access-28w48\") pod \"kas-connection-checker-685895c7c9-9m2lz\" (UID: \"8a74b249-57da-4405-af8e-ae777a44415e\") " pod="kube-system/kas-connection-checker-685895c7c9-9m2lz"
Oct 30 21:33:13.963559 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.963537    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:33:13.963670 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.963569    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 21:33:13.963755 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.963738    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 21:33:13.963817 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.963780    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 21:33:13.963869 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.963829    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 21:33:13.963869 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.963864    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:33:13.963971 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.963876    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 21:33:13.963971 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.963882    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 21:33:13.964074 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.963996    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:33:13.964074 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.964040    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:33:13.964074 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.964049    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 21:33:13.965008 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.964988    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 21:33:13.965317 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.965303    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 21:33:13.973819 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.973803    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ljjfm"]
Oct 30 21:33:13.973928 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.973914    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:13.986961 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.986945    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9trcs"]
Oct 30 21:33:13.987036 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.986970    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9m2lz"]
Oct 30 21:33:13.987036 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.986986    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cx7d4"]
Oct 30 21:33:13.987036 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.986998    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-bn8fx"]
Oct 30 21:33:13.987036 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.987012    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-qx7zq"]
Oct 30 21:33:13.987036 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.987026    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48"]
Oct 30 21:33:13.987036 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.987039    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-z4jbc"]
Oct 30 21:33:13.987240 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.987052    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-54cbfc65d5-sg48h"]
Oct 30 21:33:13.987240 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.987057    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ljjfm"
Oct 30 21:33:13.987240 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.987066    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-z8clf"]
Oct 30 21:33:13.987240 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.987079    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5kfvz"]
Oct 30 21:33:13.987240 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.987095    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ljjfm"]
Oct 30 21:33:13.987240 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.987108    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-64b9m"]
Oct 30 21:33:13.987240 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.987121    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fc9466475-jz9vh"]
Oct 30 21:33:13.987240 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.987135    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74b44b76d6-l7p76"]
Oct 30 21:33:13.987240 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.987148    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc"]
Oct 30 21:33:13.987240 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.987159    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fgpcv"]
Oct 30 21:33:13.987240 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:13.987171    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd"]
Oct 30 21:33:14.002474 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.002456    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-stats-auth\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:14.002551 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.002485    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-bound-sa-token\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:14.002551 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.002503    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-bound-sa-token\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:14.002551 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.002543    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d5f25065-34c9-4724-8dff-9ace0890d565-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qx7zq\" (UID: \"d5f25065-34c9-4724-8dff-9ace0890d565\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qx7zq"
Oct 30 21:33:14.002642 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.002590    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqsx2\" (UniqueName: \"kubernetes.io/projected/3ca424b7-c125-43ce-b837-88d1c02a05a0-kube-api-access-jqsx2\") pod \"volume-data-source-validator-58c9f5f46d-fgpcv\" (UID: \"3ca424b7-c125-43ce-b837-88d1c02a05a0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fgpcv"
Oct 30 21:33:14.002671 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.002645    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef-config\") pod \"service-ca-operator-544d749c8c-bn8fx\" (UID: \"f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bn8fx"
Oct 30 21:33:14.002701 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.002669    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 21:33:14.002701 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.002672    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42dc1fb7-b6e0-40d2-bf53-20079815b12d-installation-pull-secrets\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:14.002791 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.002697    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1196dab1-80d7-48eb-8d05-1a26ab6d0759-serving-cert\") pod \"console-operator-7cd56b8dbc-z8clf\" (UID: \"1196dab1-80d7-48eb-8d05-1a26ab6d0759\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf"
Oct 30 21:33:14.002791 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.002725    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc\" (UID: \"25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc"
Oct 30 21:33:14.002791 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.002752    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5f25065-34c9-4724-8dff-9ace0890d565-networking-console-plugin-cert podName:d5f25065-34c9-4724-8dff-9ace0890d565 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:14.502735545 +0000 UTC m=+31.829701765 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d5f25065-34c9-4724-8dff-9ace0890d565-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qx7zq" (UID: "d5f25065-34c9-4724-8dff-9ace0890d565") : secret "networking-console-plugin-cert" not found
Oct 30 21:33:14.002956 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.002782    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42dc1fb7-b6e0-40d2-bf53-20079815b12d-image-registry-private-configuration\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:14.002956 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.002844    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42dc1fb7-b6e0-40d2-bf53-20079815b12d-ca-trust-extracted\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:14.002956 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.002895    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrr6v\" (UniqueName: \"kubernetes.io/projected/1196dab1-80d7-48eb-8d05-1a26ab6d0759-kube-api-access-qrr6v\") pod \"console-operator-7cd56b8dbc-z8clf\" (UID: \"1196dab1-80d7-48eb-8d05-1a26ab6d0759\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf"
Oct 30 21:33:14.003100 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.002933    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f54f9754-2024-41b9-8f84-fc3d02a22253-tmp\") pod \"insights-operator-7c4986d449-z4jbc\" (UID: \"f54f9754-2024-41b9-8f84-fc3d02a22253\") " pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:14.003100 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.002984    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1196dab1-80d7-48eb-8d05-1a26ab6d0759-trusted-ca\") pod \"console-operator-7cd56b8dbc-z8clf\" (UID: \"1196dab1-80d7-48eb-8d05-1a26ab6d0759\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf"
Oct 30 21:33:14.003100 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.003013    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm65m\" (UniqueName: \"kubernetes.io/projected/f54f9754-2024-41b9-8f84-fc3d02a22253-kube-api-access-tm65m\") pod \"insights-operator-7c4986d449-z4jbc\" (UID: \"f54f9754-2024-41b9-8f84-fc3d02a22253\") " pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:14.003100 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.003074    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kgrcd\" (UID: \"f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd"
Oct 30 21:33:14.003482 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.003237    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-certificates\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:14.003482 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.003250    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f54f9754-2024-41b9-8f84-fc3d02a22253-tmp\") pod \"insights-operator-7c4986d449-z4jbc\" (UID: \"f54f9754-2024-41b9-8f84-fc3d02a22253\") " pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:14.003482 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.003280    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/392cc7ef-7cd2-4090-a246-0347d5af7bbf-trusted-ca\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:14.003482 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.003326    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8z8h\" (UniqueName: \"kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-kube-api-access-f8z8h\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:14.003482 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.003364    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef-config\") pod \"service-ca-operator-544d749c8c-bn8fx\" (UID: \"f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bn8fx"
Oct 30 21:33:14.003482 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.003380    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/392cc7ef-7cd2-4090-a246-0347d5af7bbf-ca-trust-extracted\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:14.003482 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.003413    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-certificates\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:14.003482 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.003473    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-tls\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:14.003872 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.003494    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f54f9754-2024-41b9-8f84-fc3d02a22253-serving-cert\") pod \"insights-operator-7c4986d449-z4jbc\" (UID: \"f54f9754-2024-41b9-8f84-fc3d02a22253\") " pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:14.003872 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.003512    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f54f9754-2024-41b9-8f84-fc3d02a22253-snapshots\") pod \"insights-operator-7c4986d449-z4jbc\" (UID: \"f54f9754-2024-41b9-8f84-fc3d02a22253\") " pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:14.003872 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.003528    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57sgv\" (UniqueName: \"kubernetes.io/projected/f84984eb-c144-4394-bb77-afc0c94b3a3a-kube-api-access-57sgv\") pod \"kas-connection-checker-685895c7c9-cx7d4\" (UID: \"f84984eb-c144-4394-bb77-afc0c94b3a3a\") " pod="kube-system/kas-connection-checker-685895c7c9-cx7d4"
Oct 30 21:33:14.003872 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.003544    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77rbg\" (UniqueName: \"kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-kube-api-access-77rbg\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:14.003872 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.003683    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wd6v\" (UniqueName: \"kubernetes.io/projected/83917e8e-50b6-4c52-b337-6f1be5a2c26a-kube-api-access-9wd6v\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:14.003872 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.003715    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2qz48\" (UID: \"9f16bd1f-a3bd-45ee-a02e-cb6b9e982327\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48"
Oct 30 21:33:14.003872 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.003745    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83917e8e-50b6-4c52-b337-6f1be5a2c26a-service-ca-bundle\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:14.003872 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.003773    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-tls\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:14.003872 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.003818    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-kgrcd\" (UID: \"f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd"
Oct 30 21:33:14.004283 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.003886    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef-serving-cert\") pod \"service-ca-operator-544d749c8c-bn8fx\" (UID: \"f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bn8fx"
Oct 30 21:33:14.004283 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.003942    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1196dab1-80d7-48eb-8d05-1a26ab6d0759-config\") pod \"console-operator-7cd56b8dbc-z8clf\" (UID: \"1196dab1-80d7-48eb-8d05-1a26ab6d0759\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf"
Oct 30 21:33:14.004283 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.003973    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-metrics-certs\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:14.004283 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.004017    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f54f9754-2024-41b9-8f84-fc3d02a22253-snapshots\") pod \"insights-operator-7c4986d449-z4jbc\" (UID: \"f54f9754-2024-41b9-8f84-fc3d02a22253\") " pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:14.004283 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.004019    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk9wl\" (UniqueName: \"kubernetes.io/projected/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-kube-api-access-hk9wl\") pod \"cluster-monitoring-operator-884576c57-2qz48\" (UID: \"9f16bd1f-a3bd-45ee-a02e-cb6b9e982327\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48"
Oct 30 21:33:14.004283 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.004065    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f54f9754-2024-41b9-8f84-fc3d02a22253-service-ca-bundle\") pod \"insights-operator-7c4986d449-z4jbc\" (UID: \"f54f9754-2024-41b9-8f84-fc3d02a22253\") " pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:14.004283 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.004091    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42dc1fb7-b6e0-40d2-bf53-20079815b12d-trusted-ca\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:14.004283 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.004121    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbqkr\" (UniqueName: \"kubernetes.io/projected/f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef-kube-api-access-fbqkr\") pod \"service-ca-operator-544d749c8c-bn8fx\" (UID: \"f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bn8fx"
Oct 30 21:33:14.004283 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.004145    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/392cc7ef-7cd2-4090-a246-0347d5af7bbf-installation-pull-secrets\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:14.004283 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.004093    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 21:33:14.004283 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.004238    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:33:14.004283 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.004251    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-metrics-certs podName:83917e8e-50b6-4c52-b337-6f1be5a2c26a nodeName:}" failed. No retries permitted until 2026-10-30 21:33:14.504218939 +0000 UTC m=+31.831185167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-metrics-certs") pod "router-default-54cbfc65d5-sg48h" (UID: "83917e8e-50b6-4c52-b337-6f1be5a2c26a") : secret "router-metrics-certs-default" not found
Oct 30 21:33:14.004283 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.004174    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2qz48\" (UID: \"9f16bd1f-a3bd-45ee-a02e-cb6b9e982327\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48"
Oct 30 21:33:14.004283 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.004283    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls podName:9f16bd1f-a3bd-45ee-a02e-cb6b9e982327 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:14.504271837 +0000 UTC m=+31.831238048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2qz48" (UID: "9f16bd1f-a3bd-45ee-a02e-cb6b9e982327") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:33:14.004990 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.004332    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f54f9754-2024-41b9-8f84-fc3d02a22253-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-z4jbc\" (UID: \"f54f9754-2024-41b9-8f84-fc3d02a22253\") " pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:14.004990 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.004381    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-default-certificate\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:14.004990 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.004413    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2mp5\" (UniqueName: \"kubernetes.io/projected/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-kube-api-access-p2mp5\") pod \"cluster-samples-operator-6ddb6c8b94-kgrcd\" (UID: \"f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd"
Oct 30 21:33:14.004990 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.004471    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgn2z\" (UniqueName: \"kubernetes.io/projected/25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245-kube-api-access-tgn2z\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc\" (UID: \"25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc"
Oct 30 21:33:14.004990 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.004501    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d5f25065-34c9-4724-8dff-9ace0890d565-nginx-conf\") pod \"networking-console-plugin-86c478bb88-qx7zq\" (UID: \"d5f25065-34c9-4724-8dff-9ace0890d565\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qx7zq"
Oct 30 21:33:14.004990 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.004543    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc\" (UID: \"25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc"
Oct 30 21:33:14.004990 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.004569    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/392cc7ef-7cd2-4090-a246-0347d5af7bbf-image-registry-private-configuration\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:14.004990 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.004581    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2qz48\" (UID: \"9f16bd1f-a3bd-45ee-a02e-cb6b9e982327\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48"
Oct 30 21:33:14.004990 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.004607    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f54f9754-2024-41b9-8f84-fc3d02a22253-service-ca-bundle\") pod \"insights-operator-7c4986d449-z4jbc\" (UID: \"f54f9754-2024-41b9-8f84-fc3d02a22253\") " pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:14.004990 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.004838    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/83917e8e-50b6-4c52-b337-6f1be5a2c26a-service-ca-bundle podName:83917e8e-50b6-4c52-b337-6f1be5a2c26a nodeName:}" failed. No retries permitted until 2026-10-30 21:33:14.504821433 +0000 UTC m=+31.831787660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/83917e8e-50b6-4c52-b337-6f1be5a2c26a-service-ca-bundle") pod "router-default-54cbfc65d5-sg48h" (UID: "83917e8e-50b6-4c52-b337-6f1be5a2c26a") : configmap references non-existent config key: service-ca.crt
Oct 30 21:33:14.005541 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.005523    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-stats-auth\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:14.005594 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.005552    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f54f9754-2024-41b9-8f84-fc3d02a22253-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-z4jbc\" (UID: \"f54f9754-2024-41b9-8f84-fc3d02a22253\") " pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:14.006054 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.006039    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d5f25065-34c9-4724-8dff-9ace0890d565-nginx-conf\") pod \"networking-console-plugin-86c478bb88-qx7zq\" (UID: \"d5f25065-34c9-4724-8dff-9ace0890d565\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qx7zq"
Oct 30 21:33:14.006340 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.006319    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f54f9754-2024-41b9-8f84-fc3d02a22253-serving-cert\") pod \"insights-operator-7c4986d449-z4jbc\" (UID: \"f54f9754-2024-41b9-8f84-fc3d02a22253\") " pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:14.006446 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.006431    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef-serving-cert\") pod \"service-ca-operator-544d749c8c-bn8fx\" (UID: \"f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bn8fx"
Oct 30 21:33:14.006997 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.006980    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-default-certificate\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:14.016819 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.016803    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-mbbvd\""
Oct 30 21:33:14.025722 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.025702    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:33:14.025799 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.025759    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-26jx5\""
Oct 30 21:33:14.034013 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.033996    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-9trcs"
Oct 30 21:33:14.039214 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.039200    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:33:14.039269 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.039246    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-flltm\""
Oct 30 21:33:14.050404 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.050392    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-9m2lz"
Oct 30 21:33:14.063340 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.063311    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm65m\" (UniqueName: \"kubernetes.io/projected/f54f9754-2024-41b9-8f84-fc3d02a22253-kube-api-access-tm65m\") pod \"insights-operator-7c4986d449-z4jbc\" (UID: \"f54f9754-2024-41b9-8f84-fc3d02a22253\") " pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:14.063449 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.063388    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wd6v\" (UniqueName: \"kubernetes.io/projected/83917e8e-50b6-4c52-b337-6f1be5a2c26a-kube-api-access-9wd6v\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:14.065026 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.065007    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57sgv\" (UniqueName: \"kubernetes.io/projected/f84984eb-c144-4394-bb77-afc0c94b3a3a-kube-api-access-57sgv\") pod \"kas-connection-checker-685895c7c9-cx7d4\" (UID: \"f84984eb-c144-4394-bb77-afc0c94b3a3a\") " pod="kube-system/kas-connection-checker-685895c7c9-cx7d4"
Oct 30 21:33:14.065125 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.065110    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:33:14.065125 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.065115    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:33:14.065188 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.065133    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:33:14.065417 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.065400    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:33:14.065460 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.065427    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:33:14.065612 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.065592    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk9wl\" (UniqueName: \"kubernetes.io/projected/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-kube-api-access-hk9wl\") pod \"cluster-monitoring-operator-884576c57-2qz48\" (UID: \"9f16bd1f-a3bd-45ee-a02e-cb6b9e982327\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48"
Oct 30 21:33:14.067380 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.067366    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbqkr\" (UniqueName: \"kubernetes.io/projected/f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef-kube-api-access-fbqkr\") pod \"service-ca-operator-544d749c8c-bn8fx\" (UID: \"f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bn8fx"
Oct 30 21:33:14.073673 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.073657    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-cx7d4"
Oct 30 21:33:14.105903 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.105867    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42dc1fb7-b6e0-40d2-bf53-20079815b12d-trusted-ca\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:14.106012 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.105925    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gw68\" (UniqueName: \"kubernetes.io/projected/b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5-kube-api-access-4gw68\") pod \"ingress-canary-ljjfm\" (UID: \"b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5\") " pod="openshift-ingress-canary/ingress-canary-ljjfm"
Oct 30 21:33:14.106012 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.105952    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/392cc7ef-7cd2-4090-a246-0347d5af7bbf-installation-pull-secrets\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:14.106012 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106000    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2mp5\" (UniqueName: \"kubernetes.io/projected/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-kube-api-access-p2mp5\") pod \"cluster-samples-operator-6ddb6c8b94-kgrcd\" (UID: \"f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd"
Oct 30 21:33:14.106125 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106030    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgn2z\" (UniqueName: \"kubernetes.io/projected/25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245-kube-api-access-tgn2z\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc\" (UID: \"25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc"
Oct 30 21:33:14.106125 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106056    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-tmp-dir\") pod \"dns-default-5kfvz\" (UID: \"2a216b8f-a750-4e1e-9ab1-ff6c89d416e5\") " pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:14.106125 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106085    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc\" (UID: \"25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc"
Oct 30 21:33:14.106125 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106111    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/392cc7ef-7cd2-4090-a246-0347d5af7bbf-image-registry-private-configuration\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:14.106240 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106138    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-bound-sa-token\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:14.106240 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106162    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-bound-sa-token\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:14.106240 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106212    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqsx2\" (UniqueName: \"kubernetes.io/projected/3ca424b7-c125-43ce-b837-88d1c02a05a0-kube-api-access-jqsx2\") pod \"volume-data-source-validator-58c9f5f46d-fgpcv\" (UID: \"3ca424b7-c125-43ce-b837-88d1c02a05a0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fgpcv"
Oct 30 21:33:14.106327 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106255    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42dc1fb7-b6e0-40d2-bf53-20079815b12d-installation-pull-secrets\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:14.106327 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106281    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6chm6\" (UniqueName: \"kubernetes.io/projected/579a0c8f-9497-4448-98ab-4686adec22cd-kube-api-access-6chm6\") pod \"network-check-source-6f78d59f99-64b9m\" (UID: \"579a0c8f-9497-4448-98ab-4686adec22cd\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-64b9m"
Oct 30 21:33:14.106327 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106305    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1196dab1-80d7-48eb-8d05-1a26ab6d0759-serving-cert\") pod \"console-operator-7cd56b8dbc-z8clf\" (UID: \"1196dab1-80d7-48eb-8d05-1a26ab6d0759\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf"
Oct 30 21:33:14.106442 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106329    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc\" (UID: \"25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc"
Oct 30 21:33:14.106442 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106375    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42dc1fb7-b6e0-40d2-bf53-20079815b12d-image-registry-private-configuration\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:14.106442 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106399    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42dc1fb7-b6e0-40d2-bf53-20079815b12d-ca-trust-extracted\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:14.106442 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106426    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrr6v\" (UniqueName: \"kubernetes.io/projected/1196dab1-80d7-48eb-8d05-1a26ab6d0759-kube-api-access-qrr6v\") pod \"console-operator-7cd56b8dbc-z8clf\" (UID: \"1196dab1-80d7-48eb-8d05-1a26ab6d0759\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf"
Oct 30 21:33:14.106557 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106450    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq5j2\" (UniqueName: \"kubernetes.io/projected/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-kube-api-access-rq5j2\") pod \"dns-default-5kfvz\" (UID: \"2a216b8f-a750-4e1e-9ab1-ff6c89d416e5\") " pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:14.106557 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106483    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1196dab1-80d7-48eb-8d05-1a26ab6d0759-trusted-ca\") pod \"console-operator-7cd56b8dbc-z8clf\" (UID: \"1196dab1-80d7-48eb-8d05-1a26ab6d0759\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf"
Oct 30 21:33:14.106557 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106513    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kgrcd\" (UID: \"f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd"
Oct 30 21:33:14.106557 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106537    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-certificates\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:14.106683 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106563    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/392cc7ef-7cd2-4090-a246-0347d5af7bbf-trusted-ca\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:14.106683 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106587    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8z8h\" (UniqueName: \"kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-kube-api-access-f8z8h\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:14.106683 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106619    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/392cc7ef-7cd2-4090-a246-0347d5af7bbf-ca-trust-extracted\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:14.106683 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106642    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-certificates\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:14.106683 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106673    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-tls\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:14.106835 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106700    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-config-volume\") pod \"dns-default-5kfvz\" (UID: \"2a216b8f-a750-4e1e-9ab1-ff6c89d416e5\") " pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:14.106835 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106741    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77rbg\" (UniqueName: \"kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-kube-api-access-77rbg\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:14.106835 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106809    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5-cert\") pod \"ingress-canary-ljjfm\" (UID: \"b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5\") " pod="openshift-ingress-canary/ingress-canary-ljjfm"
Oct 30 21:33:14.106931 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106851    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-tls\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:14.106931 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106877    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-kgrcd\" (UID: \"f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd"
Oct 30 21:33:14.106931 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106904    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1196dab1-80d7-48eb-8d05-1a26ab6d0759-config\") pod \"console-operator-7cd56b8dbc-z8clf\" (UID: \"1196dab1-80d7-48eb-8d05-1a26ab6d0759\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf"
Oct 30 21:33:14.106931 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.106927    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-metrics-tls\") pod \"dns-default-5kfvz\" (UID: \"2a216b8f-a750-4e1e-9ab1-ff6c89d416e5\") " pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:14.107900 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.107879    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42dc1fb7-b6e0-40d2-bf53-20079815b12d-trusted-ca\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:14.111932 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.108702    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-certificates\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:14.111932 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.109638    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1196dab1-80d7-48eb-8d05-1a26ab6d0759-trusted-ca\") pod \"console-operator-7cd56b8dbc-z8clf\" (UID: \"1196dab1-80d7-48eb-8d05-1a26ab6d0759\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf"
Oct 30 21:33:14.111932 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.109732    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:33:14.111932 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.109787    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-tls podName:f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf nodeName:}" failed. No retries permitted until 2026-10-30 21:33:14.609770618 +0000 UTC m=+31.936736844 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-kgrcd" (UID: "f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf") : secret "samples-operator-tls" not found
Oct 30 21:33:14.111932 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.110275    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-certificates\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:14.111932 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.111409    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/392cc7ef-7cd2-4090-a246-0347d5af7bbf-trusted-ca\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:14.111932 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.111707    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc\" (UID: \"25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc"
Oct 30 21:33:14.112506 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.112069    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:33:14.112506 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.112080    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74b44b76d6-l7p76: secret "image-registry-tls" not found
Oct 30 21:33:14.112506 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.112117    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-tls podName:392cc7ef-7cd2-4090-a246-0347d5af7bbf nodeName:}" failed. No retries permitted until 2026-10-30 21:33:14.612105699 +0000 UTC m=+31.939071924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-tls") pod "image-registry-74b44b76d6-l7p76" (UID: "392cc7ef-7cd2-4090-a246-0347d5af7bbf") : secret "image-registry-tls" not found
Oct 30 21:33:14.113285 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.112730    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-kgrcd\" (UID: \"f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd"
Oct 30 21:33:14.113285 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.112813    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:33:14.113285 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.112823    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fc9466475-jz9vh: secret "image-registry-tls" not found
Oct 30 21:33:14.113285 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.112854    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-tls podName:42dc1fb7-b6e0-40d2-bf53-20079815b12d nodeName:}" failed. No retries permitted until 2026-10-30 21:33:14.612844928 +0000 UTC m=+31.939811160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-tls") pod "image-registry-7fc9466475-jz9vh" (UID: "42dc1fb7-b6e0-40d2-bf53-20079815b12d") : secret "image-registry-tls" not found
Oct 30 21:33:14.113756 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.113540    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc\" (UID: \"25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc"
Oct 30 21:33:14.113995 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.113970    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/392cc7ef-7cd2-4090-a246-0347d5af7bbf-ca-trust-extracted\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:14.114273 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.114255    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42dc1fb7-b6e0-40d2-bf53-20079815b12d-ca-trust-extracted\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:14.115203 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.114991    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1196dab1-80d7-48eb-8d05-1a26ab6d0759-config\") pod \"console-operator-7cd56b8dbc-z8clf\" (UID: \"1196dab1-80d7-48eb-8d05-1a26ab6d0759\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf"
Oct 30 21:33:14.116395 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.116375    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/392cc7ef-7cd2-4090-a246-0347d5af7bbf-image-registry-private-configuration\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:14.116679 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.116633    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42dc1fb7-b6e0-40d2-bf53-20079815b12d-image-registry-private-configuration\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:14.116893 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.116877    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42dc1fb7-b6e0-40d2-bf53-20079815b12d-installation-pull-secrets\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:14.117575 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.117552    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/392cc7ef-7cd2-4090-a246-0347d5af7bbf-installation-pull-secrets\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:14.118162 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.118144    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1196dab1-80d7-48eb-8d05-1a26ab6d0759-serving-cert\") pod \"console-operator-7cd56b8dbc-z8clf\" (UID: \"1196dab1-80d7-48eb-8d05-1a26ab6d0759\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf"
Oct 30 21:33:14.131206 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.131186    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:14.143307 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.142615    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:33:14.143307 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.143069    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:33:14.148085 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.148061    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bn8fx"
Oct 30 21:33:14.174226 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.174199    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-bound-sa-token\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:14.175682 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.175661    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2mp5\" (UniqueName: \"kubernetes.io/projected/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-kube-api-access-p2mp5\") pod \"cluster-samples-operator-6ddb6c8b94-kgrcd\" (UID: \"f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd"
Oct 30 21:33:14.175904 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.175878    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqsx2\" (UniqueName: \"kubernetes.io/projected/3ca424b7-c125-43ce-b837-88d1c02a05a0-kube-api-access-jqsx2\") pod \"volume-data-source-validator-58c9f5f46d-fgpcv\" (UID: \"3ca424b7-c125-43ce-b837-88d1c02a05a0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fgpcv"
Oct 30 21:33:14.179119 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.179096    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrr6v\" (UniqueName: \"kubernetes.io/projected/1196dab1-80d7-48eb-8d05-1a26ab6d0759-kube-api-access-qrr6v\") pod \"console-operator-7cd56b8dbc-z8clf\" (UID: \"1196dab1-80d7-48eb-8d05-1a26ab6d0759\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf"
Oct 30 21:33:14.179198 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.179116    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8z8h\" (UniqueName: \"kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-kube-api-access-f8z8h\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:14.179249 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.179208    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77rbg\" (UniqueName: \"kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-kube-api-access-77rbg\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:14.180053 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.180009    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgn2z\" (UniqueName: \"kubernetes.io/projected/25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245-kube-api-access-tgn2z\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc\" (UID: \"25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc"
Oct 30 21:33:14.181306 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.181186    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-bound-sa-token\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:14.196167 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.195835    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-qlt7z\""
Oct 30 21:33:14.196167 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.195928    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-76thg\""
Oct 30 21:33:14.196167 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.195957    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:33:14.196167 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.196139    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:33:14.207422 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.207401    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gw68\" (UniqueName: \"kubernetes.io/projected/b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5-kube-api-access-4gw68\") pod \"ingress-canary-ljjfm\" (UID: \"b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5\") " pod="openshift-ingress-canary/ingress-canary-ljjfm"
Oct 30 21:33:14.207526 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.207442    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-tmp-dir\") pod \"dns-default-5kfvz\" (UID: \"2a216b8f-a750-4e1e-9ab1-ff6c89d416e5\") " pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:14.207526 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.207490    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6chm6\" (UniqueName: \"kubernetes.io/projected/579a0c8f-9497-4448-98ab-4686adec22cd-kube-api-access-6chm6\") pod \"network-check-source-6f78d59f99-64b9m\" (UID: \"579a0c8f-9497-4448-98ab-4686adec22cd\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-64b9m"
Oct 30 21:33:14.207526 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.207512    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq5j2\" (UniqueName: \"kubernetes.io/projected/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-kube-api-access-rq5j2\") pod \"dns-default-5kfvz\" (UID: \"2a216b8f-a750-4e1e-9ab1-ff6c89d416e5\") " pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:14.207636 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.207561    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-config-volume\") pod \"dns-default-5kfvz\" (UID: \"2a216b8f-a750-4e1e-9ab1-ff6c89d416e5\") " pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:14.207636 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.207597    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5-cert\") pod \"ingress-canary-ljjfm\" (UID: \"b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5\") " pod="openshift-ingress-canary/ingress-canary-ljjfm"
Oct 30 21:33:14.207636 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.207628    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-metrics-tls\") pod \"dns-default-5kfvz\" (UID: \"2a216b8f-a750-4e1e-9ab1-ff6c89d416e5\") " pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:14.207734 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.207726    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:33:14.207775 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.207769    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-metrics-tls podName:2a216b8f-a750-4e1e-9ab1-ff6c89d416e5 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:14.707757391 +0000 UTC m=+32.034723615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-metrics-tls") pod "dns-default-5kfvz" (UID: "2a216b8f-a750-4e1e-9ab1-ff6c89d416e5") : secret "dns-default-metrics-tls" not found
Oct 30 21:33:14.208175 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.208158    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-tmp-dir\") pod \"dns-default-5kfvz\" (UID: \"2a216b8f-a750-4e1e-9ab1-ff6c89d416e5\") " pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:14.208799 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.208782    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-config-volume\") pod \"dns-default-5kfvz\" (UID: \"2a216b8f-a750-4e1e-9ab1-ff6c89d416e5\") " pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:14.211426 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.211379    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5-cert\") pod \"ingress-canary-ljjfm\" (UID: \"b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5\") " pod="openshift-ingress-canary/ingress-canary-ljjfm"
Oct 30 21:33:14.217880 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.216866    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc"
Oct 30 21:33:14.242047 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.241577    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fgpcv"
Oct 30 21:33:14.269407 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.268757    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6chm6\" (UniqueName: \"kubernetes.io/projected/579a0c8f-9497-4448-98ab-4686adec22cd-kube-api-access-6chm6\") pod \"network-check-source-6f78d59f99-64b9m\" (UID: \"579a0c8f-9497-4448-98ab-4686adec22cd\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-64b9m"
Oct 30 21:33:14.278319 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.278259    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gw68\" (UniqueName: \"kubernetes.io/projected/b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5-kube-api-access-4gw68\") pod \"ingress-canary-ljjfm\" (UID: \"b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5\") " pod="openshift-ingress-canary/ingress-canary-ljjfm"
Oct 30 21:33:14.280434 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.280213    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq5j2\" (UniqueName: \"kubernetes.io/projected/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-kube-api-access-rq5j2\") pod \"dns-default-5kfvz\" (UID: \"2a216b8f-a750-4e1e-9ab1-ff6c89d416e5\") " pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:14.330048 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.330021    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-64b9m"
Oct 30 21:33:14.348054 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.348031    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ljjfm"
Oct 30 21:33:14.352238 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.352218    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-h5xkl"
Oct 30 21:33:14.352393 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.352381    2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:33:14.352749 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.352735    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h5xkl"
Oct 30 21:33:14.468551 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.468487    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf"
Oct 30 21:33:14.486283 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.486252    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9m2lz"]
Oct 30 21:33:14.487095 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.487068    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cx7d4"]
Oct 30 21:33:14.488618 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.488597    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-9trcs"]
Oct 30 21:33:14.491141 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:33:14.491108    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf84984eb_c144_4394_bb77_afc0c94b3a3a.slice/crio-1b59cf8e851f9e4d60b0d55c5c8974801b568297de2f4f5b764aa9399df2eb83 WatchSource:0}: Error finding container 1b59cf8e851f9e4d60b0d55c5c8974801b568297de2f4f5b764aa9399df2eb83: Status 404 returned error can't find the container with id 1b59cf8e851f9e4d60b0d55c5c8974801b568297de2f4f5b764aa9399df2eb83
Oct 30 21:33:14.491466 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:33:14.491441    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a74b249_57da_4405_af8e_ae777a44415e.slice/crio-4d36ec40de3b72acea76d6288acfc84460e0929e3022b70116f7c478ecabc504 WatchSource:0}: Error finding container 4d36ec40de3b72acea76d6288acfc84460e0929e3022b70116f7c478ecabc504: Status 404 returned error can't find the container with id 4d36ec40de3b72acea76d6288acfc84460e0929e3022b70116f7c478ecabc504
Oct 30 21:33:14.493005 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:33:14.492986    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod364111d6_7289_47be_8cf0_c10c7b6fbcdc.slice/crio-426c2d37a383fc0ad646fefef08f863879895c806c0dd0514b0e746ec625f730 WatchSource:0}: Error finding container 426c2d37a383fc0ad646fefef08f863879895c806c0dd0514b0e746ec625f730: Status 404 returned error can't find the container with id 426c2d37a383fc0ad646fefef08f863879895c806c0dd0514b0e746ec625f730
Oct 30 21:33:14.516432 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.516402    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83917e8e-50b6-4c52-b337-6f1be5a2c26a-service-ca-bundle\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:14.516552 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.516473    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-metrics-certs\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:14.516552 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.516515    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2qz48\" (UID: \"9f16bd1f-a3bd-45ee-a02e-cb6b9e982327\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48"
Oct 30 21:33:14.516663 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.516608    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d5f25065-34c9-4724-8dff-9ace0890d565-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qx7zq\" (UID: \"d5f25065-34c9-4724-8dff-9ace0890d565\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qx7zq"
Oct 30 21:33:14.516768 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.516749    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 21:33:14.516840 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.516820    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5f25065-34c9-4724-8dff-9ace0890d565-networking-console-plugin-cert podName:d5f25065-34c9-4724-8dff-9ace0890d565 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:15.516801761 +0000 UTC m=+32.843767989 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d5f25065-34c9-4724-8dff-9ace0890d565-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qx7zq" (UID: "d5f25065-34c9-4724-8dff-9ace0890d565") : secret "networking-console-plugin-cert" not found
Oct 30 21:33:14.517330 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.517295    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 21:33:14.517438 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.517372    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-metrics-certs podName:83917e8e-50b6-4c52-b337-6f1be5a2c26a nodeName:}" failed. No retries permitted until 2026-10-30 21:33:15.517337871 +0000 UTC m=+32.844304096 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-metrics-certs") pod "router-default-54cbfc65d5-sg48h" (UID: "83917e8e-50b6-4c52-b337-6f1be5a2c26a") : secret "router-metrics-certs-default" not found
Oct 30 21:33:14.517438 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.517434    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:33:14.517547 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.517480    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/83917e8e-50b6-4c52-b337-6f1be5a2c26a-service-ca-bundle podName:83917e8e-50b6-4c52-b337-6f1be5a2c26a nodeName:}" failed. No retries permitted until 2026-10-30 21:33:15.517462543 +0000 UTC m=+32.844428769 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/83917e8e-50b6-4c52-b337-6f1be5a2c26a-service-ca-bundle") pod "router-default-54cbfc65d5-sg48h" (UID: "83917e8e-50b6-4c52-b337-6f1be5a2c26a") : configmap references non-existent config key: service-ca.crt
Oct 30 21:33:14.517547 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.517503    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls podName:9f16bd1f-a3bd-45ee-a02e-cb6b9e982327 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:15.517494219 +0000 UTC m=+32.844460461 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2qz48" (UID: "9f16bd1f-a3bd-45ee-a02e-cb6b9e982327") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:33:14.556013 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.555930    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-z4jbc"]
Oct 30 21:33:14.562230 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:33:14.562038    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf54f9754_2024_41b9_8f84_fc3d02a22253.slice/crio-0950186ea5083c75b1b9fae181e158056f873b387340151e7be3e063325afb53 WatchSource:0}: Error finding container 0950186ea5083c75b1b9fae181e158056f873b387340151e7be3e063325afb53: Status 404 returned error can't find the container with id 0950186ea5083c75b1b9fae181e158056f873b387340151e7be3e063325afb53
Oct 30 21:33:14.573600 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.573534    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-bn8fx"]
Oct 30 21:33:14.583926 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:33:14.583892    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d73b67_bdd0_4ab1_a8b2_0395e8ff73ef.slice/crio-916d64cb3b64d505ae30dc11ed921fcc439c6387395ff358fa55664fba9405af WatchSource:0}: Error finding container 916d64cb3b64d505ae30dc11ed921fcc439c6387395ff358fa55664fba9405af: Status 404 returned error can't find the container with id 916d64cb3b64d505ae30dc11ed921fcc439c6387395ff358fa55664fba9405af
Oct 30 21:33:14.613325 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.613236    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc"]
Oct 30 21:33:14.615919 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:33:14.615892    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b8da2c_9c9d_41ab_8cf4_1a6d7f5e9245.slice/crio-2e73812d15ea38983aa08ff34298051b5b83a7ac0e2b02aa7ded043c23fddc62 WatchSource:0}: Error finding container 2e73812d15ea38983aa08ff34298051b5b83a7ac0e2b02aa7ded043c23fddc62: Status 404 returned error can't find the container with id 2e73812d15ea38983aa08ff34298051b5b83a7ac0e2b02aa7ded043c23fddc62
Oct 30 21:33:14.617485 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.617457    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kgrcd\" (UID: \"f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd"
Oct 30 21:33:14.617576 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.617516    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-tls\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:14.617627 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.617571    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-tls\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:14.617627 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.617597    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:33:14.617723 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.617644    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:33:14.617723 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.617654    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-tls podName:f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf nodeName:}" failed. No retries permitted until 2026-10-30 21:33:15.617635871 +0000 UTC m=+32.944602100 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-kgrcd" (UID: "f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf") : secret "samples-operator-tls" not found
Oct 30 21:33:14.617723 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.617656    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74b44b76d6-l7p76: secret "image-registry-tls" not found
Oct 30 21:33:14.617723 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.617689    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:33:14.617723 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.617697    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-tls podName:392cc7ef-7cd2-4090-a246-0347d5af7bbf nodeName:}" failed. No retries permitted until 2026-10-30 21:33:15.617686624 +0000 UTC m=+32.944652836 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-tls") pod "image-registry-74b44b76d6-l7p76" (UID: "392cc7ef-7cd2-4090-a246-0347d5af7bbf") : secret "image-registry-tls" not found
Oct 30 21:33:14.617723 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.617699    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fc9466475-jz9vh: secret "image-registry-tls" not found
Oct 30 21:33:14.618015 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.617736    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-tls podName:42dc1fb7-b6e0-40d2-bf53-20079815b12d nodeName:}" failed. No retries permitted until 2026-10-30 21:33:15.617724737 +0000 UTC m=+32.944690948 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-tls") pod "image-registry-7fc9466475-jz9vh" (UID: "42dc1fb7-b6e0-40d2-bf53-20079815b12d") : secret "image-registry-tls" not found
Oct 30 21:33:14.641694 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.641671    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fgpcv"]
Oct 30 21:33:14.644219 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:33:14.644196    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ca424b7_c125_43ce_b837_88d1c02a05a0.slice/crio-add53b6f3bed35e52ef02565ebc5fe9d113ee6f5773c9203f1ff69f581149e0b WatchSource:0}: Error finding container add53b6f3bed35e52ef02565ebc5fe9d113ee6f5773c9203f1ff69f581149e0b: Status 404 returned error can't find the container with id add53b6f3bed35e52ef02565ebc5fe9d113ee6f5773c9203f1ff69f581149e0b
Oct 30 21:33:14.718790 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.718727    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-metrics-tls\") pod \"dns-default-5kfvz\" (UID: \"2a216b8f-a750-4e1e-9ab1-ff6c89d416e5\") " pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:14.718919 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.718882    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:33:14.718962 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:14.718935    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-metrics-tls podName:2a216b8f-a750-4e1e-9ab1-ff6c89d416e5 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:15.718920648 +0000 UTC m=+33.045886873 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-metrics-tls") pod "dns-default-5kfvz" (UID: "2a216b8f-a750-4e1e-9ab1-ff6c89d416e5") : secret "dns-default-metrics-tls" not found
Oct 30 21:33:14.725831 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.725812    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-64b9m"]
Oct 30 21:33:14.727775 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:33:14.727752    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod579a0c8f_9497_4448_98ab_4686adec22cd.slice/crio-ddb24eb1d1c6999a890779b2b932c9862c7f27eed0cd17007ad7d11bec1993f1 WatchSource:0}: Error finding container ddb24eb1d1c6999a890779b2b932c9862c7f27eed0cd17007ad7d11bec1993f1: Status 404 returned error can't find the container with id ddb24eb1d1c6999a890779b2b932c9862c7f27eed0cd17007ad7d11bec1993f1
Oct 30 21:33:14.752942 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.752891    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ljjfm"]
Oct 30 21:33:14.755101 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:33:14.755080    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7252dc2_b8ae_41f4_b7d8_c0ca9986f8e5.slice/crio-c3be87b2dc812051283450885198270cb1f6be3b7a2a5ed7a65389032a4fe0f8 WatchSource:0}: Error finding container c3be87b2dc812051283450885198270cb1f6be3b7a2a5ed7a65389032a4fe0f8: Status 404 returned error can't find the container with id c3be87b2dc812051283450885198270cb1f6be3b7a2a5ed7a65389032a4fe0f8
Oct 30 21:33:14.885465 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:14.885438    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-z8clf"]
Oct 30 21:33:14.888496 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:33:14.888471    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1196dab1_80d7_48eb_8d05_1a26ab6d0759.slice/crio-27149485ed43e2d13c16e47e539a93d2b28a9cb674c3d1bb83aedb22ee74a72e WatchSource:0}: Error finding container 27149485ed43e2d13c16e47e539a93d2b28a9cb674c3d1bb83aedb22ee74a72e: Status 404 returned error can't find the container with id 27149485ed43e2d13c16e47e539a93d2b28a9cb674c3d1bb83aedb22ee74a72e
Oct 30 21:33:15.142548 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:15.142479    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:33:15.195533 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:15.195508    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-cprz9\""
Oct 30 21:33:15.348578 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:15.348545    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-64b9m" event={"ID":"579a0c8f-9497-4448-98ab-4686adec22cd","Type":"ContainerStarted","Data":"ddb24eb1d1c6999a890779b2b932c9862c7f27eed0cd17007ad7d11bec1993f1"}
Oct 30 21:33:15.349575 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:15.349547    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc" event={"ID":"25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245","Type":"ContainerStarted","Data":"2e73812d15ea38983aa08ff34298051b5b83a7ac0e2b02aa7ded043c23fddc62"}
Oct 30 21:33:15.350945 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:15.350916    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-9m2lz" event={"ID":"8a74b249-57da-4405-af8e-ae777a44415e","Type":"ContainerStarted","Data":"42f5c667d4b9025b3c38ba45e7ef71c86a79f448f8d464073889f4bb6d9a852f"}
Oct 30 21:33:15.351036 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:15.350961    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-9m2lz" event={"ID":"8a74b249-57da-4405-af8e-ae777a44415e","Type":"ContainerStarted","Data":"4d36ec40de3b72acea76d6288acfc84460e0929e3022b70116f7c478ecabc504"}
Oct 30 21:33:15.352379 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:15.352356    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-cx7d4" event={"ID":"f84984eb-c144-4394-bb77-afc0c94b3a3a","Type":"ContainerStarted","Data":"dea78464fda6b491650722070c13c846dfe49e47f1615f03cd3703b8488cab1e"}
Oct 30 21:33:15.352467 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:15.352397    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-cx7d4" event={"ID":"f84984eb-c144-4394-bb77-afc0c94b3a3a","Type":"ContainerStarted","Data":"1b59cf8e851f9e4d60b0d55c5c8974801b568297de2f4f5b764aa9399df2eb83"}
Oct 30 21:33:15.353779 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:15.353756    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-9trcs" event={"ID":"364111d6-7289-47be-8cf0-c10c7b6fbcdc","Type":"ContainerStarted","Data":"dad1c2b77d2f2e5d3def780c3ccd67c8cc8fac620a4b22ea23ff0e39a5064451"}
Oct 30 21:33:15.353857 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:15.353797    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-9trcs" event={"ID":"364111d6-7289-47be-8cf0-c10c7b6fbcdc","Type":"ContainerStarted","Data":"426c2d37a383fc0ad646fefef08f863879895c806c0dd0514b0e746ec625f730"}
Oct 30 21:33:15.354988 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:15.354971    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ljjfm" event={"ID":"b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5","Type":"ContainerStarted","Data":"c3be87b2dc812051283450885198270cb1f6be3b7a2a5ed7a65389032a4fe0f8"}
Oct 30 21:33:15.359834 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:15.356127    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bn8fx" event={"ID":"f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef","Type":"ContainerStarted","Data":"916d64cb3b64d505ae30dc11ed921fcc439c6387395ff358fa55664fba9405af"}
Oct 30 21:33:15.361515 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:15.361404    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf" event={"ID":"1196dab1-80d7-48eb-8d05-1a26ab6d0759","Type":"ContainerStarted","Data":"27149485ed43e2d13c16e47e539a93d2b28a9cb674c3d1bb83aedb22ee74a72e"}
Oct 30 21:33:15.362554 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:15.362537    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fgpcv" event={"ID":"3ca424b7-c125-43ce-b837-88d1c02a05a0","Type":"ContainerStarted","Data":"add53b6f3bed35e52ef02565ebc5fe9d113ee6f5773c9203f1ff69f581149e0b"}
Oct 30 21:33:15.363674 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:15.363653    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-z4jbc" event={"ID":"f54f9754-2024-41b9-8f84-fc3d02a22253","Type":"ContainerStarted","Data":"0950186ea5083c75b1b9fae181e158056f873b387340151e7be3e063325afb53"}
Oct 30 21:33:15.526756 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:15.526668    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83917e8e-50b6-4c52-b337-6f1be5a2c26a-service-ca-bundle\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:15.526756 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:15.526742    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-metrics-certs\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:15.526966 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:15.526774    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2qz48\" (UID: \"9f16bd1f-a3bd-45ee-a02e-cb6b9e982327\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48"
Oct 30 21:33:15.526966 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:15.526840    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d5f25065-34c9-4724-8dff-9ace0890d565-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qx7zq\" (UID: \"d5f25065-34c9-4724-8dff-9ace0890d565\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qx7zq"
Oct 30 21:33:15.526966 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:15.526865    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/83917e8e-50b6-4c52-b337-6f1be5a2c26a-service-ca-bundle podName:83917e8e-50b6-4c52-b337-6f1be5a2c26a nodeName:}" failed. No retries permitted until 2026-10-30 21:33:17.526842581 +0000 UTC m=+34.853808805 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/83917e8e-50b6-4c52-b337-6f1be5a2c26a-service-ca-bundle") pod "router-default-54cbfc65d5-sg48h" (UID: "83917e8e-50b6-4c52-b337-6f1be5a2c26a") : configmap references non-existent config key: service-ca.crt
Oct 30 21:33:15.526966 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:15.526914    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 21:33:15.526966 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:15.526964    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5f25065-34c9-4724-8dff-9ace0890d565-networking-console-plugin-cert podName:d5f25065-34c9-4724-8dff-9ace0890d565 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:17.526949231 +0000 UTC m=+34.853915458 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d5f25065-34c9-4724-8dff-9ace0890d565-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qx7zq" (UID: "d5f25065-34c9-4724-8dff-9ace0890d565") : secret "networking-console-plugin-cert" not found
Oct 30 21:33:15.527447 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:15.527425    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 21:33:15.527577 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:15.527464    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-metrics-certs podName:83917e8e-50b6-4c52-b337-6f1be5a2c26a nodeName:}" failed. No retries permitted until 2026-10-30 21:33:17.527455142 +0000 UTC m=+34.854421384 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-metrics-certs") pod "router-default-54cbfc65d5-sg48h" (UID: "83917e8e-50b6-4c52-b337-6f1be5a2c26a") : secret "router-metrics-certs-default" not found
Oct 30 21:33:15.527821 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:15.527801    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:33:15.527913 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:15.527852    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls podName:9f16bd1f-a3bd-45ee-a02e-cb6b9e982327 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:17.527837899 +0000 UTC m=+34.854804128 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2qz48" (UID: "9f16bd1f-a3bd-45ee-a02e-cb6b9e982327") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:33:15.628399 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:15.628364    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kgrcd\" (UID: \"f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd"
Oct 30 21:33:15.628570 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:15.628429    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-tls\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:15.628570 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:15.628494    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-tls\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:15.628570 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:15.628505    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:33:15.628570 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:15.628555    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:33:15.628739 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:15.628574    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74b44b76d6-l7p76: secret "image-registry-tls" not found
Oct 30 21:33:15.628739 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:15.628577    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-tls podName:f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf nodeName:}" failed. No retries permitted until 2026-10-30 21:33:17.628556048 +0000 UTC m=+34.955522263 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-kgrcd" (UID: "f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf") : secret "samples-operator-tls" not found
Oct 30 21:33:15.628739 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:15.628598    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:33:15.628739 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:15.628611    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fc9466475-jz9vh: secret "image-registry-tls" not found
Oct 30 21:33:15.628739 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:15.628628    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-tls podName:392cc7ef-7cd2-4090-a246-0347d5af7bbf nodeName:}" failed. No retries permitted until 2026-10-30 21:33:17.628608978 +0000 UTC m=+34.955575192 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-tls") pod "image-registry-74b44b76d6-l7p76" (UID: "392cc7ef-7cd2-4090-a246-0347d5af7bbf") : secret "image-registry-tls" not found
Oct 30 21:33:15.628739 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:15.628649    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-tls podName:42dc1fb7-b6e0-40d2-bf53-20079815b12d nodeName:}" failed. No retries permitted until 2026-10-30 21:33:17.628636268 +0000 UTC m=+34.955602482 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-tls") pod "image-registry-7fc9466475-jz9vh" (UID: "42dc1fb7-b6e0-40d2-bf53-20079815b12d") : secret "image-registry-tls" not found
Oct 30 21:33:15.730144 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:15.729795    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-metrics-tls\") pod \"dns-default-5kfvz\" (UID: \"2a216b8f-a750-4e1e-9ab1-ff6c89d416e5\") " pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:15.730144 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:15.730050    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:33:15.730144 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:15.730111    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-metrics-tls podName:2a216b8f-a750-4e1e-9ab1-ff6c89d416e5 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:17.730093551 +0000 UTC m=+35.057059785 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-metrics-tls") pod "dns-default-5kfvz" (UID: "2a216b8f-a750-4e1e-9ab1-ff6c89d416e5") : secret "dns-default-metrics-tls" not found
Oct 30 21:33:16.097601 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:16.097539    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-9trcs" podStartSLOduration=428.097518778 podStartE2EDuration="7m8.097518778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:15.977632778 +0000 UTC m=+33.304599014" watchObservedRunningTime="2026-10-30 21:33:16.097518778 +0000 UTC m=+33.424485014"
Oct 30 21:33:16.098190 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:16.098152    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-9m2lz" podStartSLOduration=428.09814141 podStartE2EDuration="7m8.09814141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:16.093475472 +0000 UTC m=+33.420441708" watchObservedRunningTime="2026-10-30 21:33:16.09814141 +0000 UTC m=+33.425107645"
Oct 30 21:33:16.135181 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:16.135142    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs\") pod \"network-metrics-daemon-bxbq6\" (UID: \"032b9335-657e-4a86-aa6d-4ef46163014a\") " pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:33:16.135435 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:16.135325    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-original-pull-secret\") pod \"global-pull-secret-syncer-2z8sh\" (UID: \"2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49\") " pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:33:16.135989 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:16.135969    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:33:16.136077 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:16.136040    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs podName:032b9335-657e-4a86-aa6d-4ef46163014a nodeName:}" failed. No retries permitted until 2026-10-30 21:33:48.136021449 +0000 UTC m=+65.462987662 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs") pod "network-metrics-daemon-bxbq6" (UID: "032b9335-657e-4a86-aa6d-4ef46163014a") : secret "metrics-daemon-secret" not found
Oct 30 21:33:16.154493 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:16.154459    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-original-pull-secret\") pod \"global-pull-secret-syncer-2z8sh\" (UID: \"2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49\") " pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:33:16.214492 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:16.214427    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-cx7d4" podStartSLOduration=428.214408626 podStartE2EDuration="7m8.214408626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:16.213790481 +0000 UTC m=+33.540756714" watchObservedRunningTime="2026-10-30 21:33:16.214408626 +0000 UTC m=+33.541374860"
Oct 30 21:33:16.236515 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:16.236479    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcz4j\" (UniqueName: \"kubernetes.io/projected/e1a448f1-e180-4fbf-9923-fac0b9dbec74-kube-api-access-xcz4j\") pod \"network-check-target-spx9v\" (UID: \"e1a448f1-e180-4fbf-9923-fac0b9dbec74\") " pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:33:16.247817 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:16.247788    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcz4j\" (UniqueName: \"kubernetes.io/projected/e1a448f1-e180-4fbf-9923-fac0b9dbec74-kube-api-access-xcz4j\") pod \"network-check-target-spx9v\" (UID: \"e1a448f1-e180-4fbf-9923-fac0b9dbec74\") " pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:33:16.275600 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:16.275507    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:33:16.390063 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:16.390007    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:33:16.703973 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:16.703852    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-2z8sh"]
Oct 30 21:33:16.711969 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:33:16.711940    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a95f3f4_cfa1_4ae6_b91b_bc20f1ccbb49.slice/crio-88bd934e2000ef20455885675bf53af8a22f08bf75971d6fae1312c882bb3c38 WatchSource:0}: Error finding container 88bd934e2000ef20455885675bf53af8a22f08bf75971d6fae1312c882bb3c38: Status 404 returned error can't find the container with id 88bd934e2000ef20455885675bf53af8a22f08bf75971d6fae1312c882bb3c38
Oct 30 21:33:16.818874 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:16.818841    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-spx9v"]
Oct 30 21:33:17.375879 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:17.375841    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-2z8sh" event={"ID":"2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49","Type":"ContainerStarted","Data":"88bd934e2000ef20455885675bf53af8a22f08bf75971d6fae1312c882bb3c38"}
Oct 30 21:33:17.377583 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:17.377478    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-spx9v" event={"ID":"e1a448f1-e180-4fbf-9923-fac0b9dbec74","Type":"ContainerStarted","Data":"ab1225b01af59fa63aff8166519a21119d51187887ff21d6c6fe4aedb3fbd04d"}
Oct 30 21:33:17.551583 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:17.550777    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83917e8e-50b6-4c52-b337-6f1be5a2c26a-service-ca-bundle\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:17.551583 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:17.550850    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-metrics-certs\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:17.551583 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:17.550889    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2qz48\" (UID: \"9f16bd1f-a3bd-45ee-a02e-cb6b9e982327\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48"
Oct 30 21:33:17.551583 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:17.550944    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d5f25065-34c9-4724-8dff-9ace0890d565-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qx7zq\" (UID: \"d5f25065-34c9-4724-8dff-9ace0890d565\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qx7zq"
Oct 30 21:33:17.551583 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:17.551166    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 21:33:17.551583 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:17.551227    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5f25065-34c9-4724-8dff-9ace0890d565-networking-console-plugin-cert podName:d5f25065-34c9-4724-8dff-9ace0890d565 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:21.551209477 +0000 UTC m=+38.878175691 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d5f25065-34c9-4724-8dff-9ace0890d565-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qx7zq" (UID: "d5f25065-34c9-4724-8dff-9ace0890d565") : secret "networking-console-plugin-cert" not found
Oct 30 21:33:17.551583 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:17.551364    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/83917e8e-50b6-4c52-b337-6f1be5a2c26a-service-ca-bundle podName:83917e8e-50b6-4c52-b337-6f1be5a2c26a nodeName:}" failed. No retries permitted until 2026-10-30 21:33:21.551335543 +0000 UTC m=+38.878301756 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/83917e8e-50b6-4c52-b337-6f1be5a2c26a-service-ca-bundle") pod "router-default-54cbfc65d5-sg48h" (UID: "83917e8e-50b6-4c52-b337-6f1be5a2c26a") : configmap references non-existent config key: service-ca.crt
Oct 30 21:33:17.551583 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:17.551430    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 21:33:17.551583 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:17.551462    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-metrics-certs podName:83917e8e-50b6-4c52-b337-6f1be5a2c26a nodeName:}" failed. No retries permitted until 2026-10-30 21:33:21.55145245 +0000 UTC m=+38.878418664 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-metrics-certs") pod "router-default-54cbfc65d5-sg48h" (UID: "83917e8e-50b6-4c52-b337-6f1be5a2c26a") : secret "router-metrics-certs-default" not found
Oct 30 21:33:17.551583 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:17.551522    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:33:17.551583 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:17.551557    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls podName:9f16bd1f-a3bd-45ee-a02e-cb6b9e982327 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:21.551546923 +0000 UTC m=+38.878513138 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2qz48" (UID: "9f16bd1f-a3bd-45ee-a02e-cb6b9e982327") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:33:17.652269 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:17.651782    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kgrcd\" (UID: \"f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd"
Oct 30 21:33:17.652269 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:17.651829    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-tls\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:17.652269 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:17.651887    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-tls\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:17.652269 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:17.652025    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:33:17.652269 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:17.652036    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fc9466475-jz9vh: secret "image-registry-tls" not found
Oct 30 21:33:17.652269 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:17.652088    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-tls podName:42dc1fb7-b6e0-40d2-bf53-20079815b12d nodeName:}" failed. No retries permitted until 2026-10-30 21:33:21.652071428 +0000 UTC m=+38.979037645 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-tls") pod "image-registry-7fc9466475-jz9vh" (UID: "42dc1fb7-b6e0-40d2-bf53-20079815b12d") : secret "image-registry-tls" not found
Oct 30 21:33:17.652269 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:17.652152    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:33:17.652269 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:17.652180    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-tls podName:f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf nodeName:}" failed. No retries permitted until 2026-10-30 21:33:21.65216808 +0000 UTC m=+38.979134291 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-kgrcd" (UID: "f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf") : secret "samples-operator-tls" not found
Oct 30 21:33:17.652269 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:17.652219    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:33:17.652269 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:17.652226    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74b44b76d6-l7p76: secret "image-registry-tls" not found
Oct 30 21:33:17.652269 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:17.652245    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-tls podName:392cc7ef-7cd2-4090-a246-0347d5af7bbf nodeName:}" failed. No retries permitted until 2026-10-30 21:33:21.652238165 +0000 UTC m=+38.979204376 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-tls") pod "image-registry-74b44b76d6-l7p76" (UID: "392cc7ef-7cd2-4090-a246-0347d5af7bbf") : secret "image-registry-tls" not found
Oct 30 21:33:17.752564 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:17.752529    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-metrics-tls\") pod \"dns-default-5kfvz\" (UID: \"2a216b8f-a750-4e1e-9ab1-ff6c89d416e5\") " pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:17.752739 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:17.752687    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:33:17.752800 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:17.752744    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-metrics-tls podName:2a216b8f-a750-4e1e-9ab1-ff6c89d416e5 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:21.752727373 +0000 UTC m=+39.079693601 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-metrics-tls") pod "dns-default-5kfvz" (UID: "2a216b8f-a750-4e1e-9ab1-ff6c89d416e5") : secret "dns-default-metrics-tls" not found
Oct 30 21:33:18.385774 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:18.385724    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fgpcv" event={"ID":"3ca424b7-c125-43ce-b837-88d1c02a05a0","Type":"ContainerStarted","Data":"7f9b48fd3e3881491850bff077bfe16a84f4b4f67447bbaa1f812de6fbdd1e5a"}
Oct 30 21:33:20.401374 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:20.400794    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/0.log"
Oct 30 21:33:20.401374 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:20.400843    2572 generic.go:356] "Generic (PLEG): container finished" podID="1196dab1-80d7-48eb-8d05-1a26ab6d0759" containerID="9de56ee88365f82422d5340ff90fc464bc6003f132b466813c94ccd874887559" exitCode=255
Oct 30 21:33:20.401374 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:20.400928    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf" event={"ID":"1196dab1-80d7-48eb-8d05-1a26ab6d0759","Type":"ContainerDied","Data":"9de56ee88365f82422d5340ff90fc464bc6003f132b466813c94ccd874887559"}
Oct 30 21:33:20.401374 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:20.401145    2572 scope.go:122] "RemoveContainer" containerID="9de56ee88365f82422d5340ff90fc464bc6003f132b466813c94ccd874887559"
Oct 30 21:33:20.404299 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:20.403754    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc" event={"ID":"25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245","Type":"ContainerStarted","Data":"eee1afe06d79357285df9d6fe0e1ca4f382b777055128461b08ab6b0da5ba34a"}
Oct 30 21:33:20.516922 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:20.515452    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fgpcv" podStartSLOduration=400.621974065 podStartE2EDuration="6m43.515433088s" totalImagesPullingTime="2.893459023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:18.502236934 +0000 UTC m=+35.829203167" watchObservedRunningTime="2026-10-30 21:33:20.515433088 +0000 UTC m=+37.842399324"
Oct 30 21:33:20.631429 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:20.631372    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc" podStartSLOduration=398.474756913 podStartE2EDuration="6m43.63133637s" totalImagesPullingTime="5.156579457s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:20.630494731 +0000 UTC m=+37.957460966" watchObservedRunningTime="2026-10-30 21:33:20.63133637 +0000 UTC m=+37.958302620"
Oct 30 21:33:21.410431 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:21.410400    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 21:33:21.411182 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:21.411158    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/0.log"
Oct 30 21:33:21.411303 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:21.411204    2572 generic.go:356] "Generic (PLEG): container finished" podID="1196dab1-80d7-48eb-8d05-1a26ab6d0759" containerID="d036f501bf39148b5ef1dfc318a42d74c69b78f8ea89630deb12c0797ede2521" exitCode=255
Oct 30 21:33:21.411303 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:21.411293    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf" event={"ID":"1196dab1-80d7-48eb-8d05-1a26ab6d0759","Type":"ContainerDied","Data":"d036f501bf39148b5ef1dfc318a42d74c69b78f8ea89630deb12c0797ede2521"}
Oct 30 21:33:21.411462 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:21.411389    2572 scope.go:122] "RemoveContainer" containerID="9de56ee88365f82422d5340ff90fc464bc6003f132b466813c94ccd874887559"
Oct 30 21:33:21.412022 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:21.411576    2572 scope.go:122] "RemoveContainer" containerID="d036f501bf39148b5ef1dfc318a42d74c69b78f8ea89630deb12c0797ede2521"
Oct 30 21:33:21.412022 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:21.411792    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-z8clf_openshift-console-operator(1196dab1-80d7-48eb-8d05-1a26ab6d0759)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf" podUID="1196dab1-80d7-48eb-8d05-1a26ab6d0759"
Oct 30 21:33:21.413712 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:21.413691    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-z4jbc" event={"ID":"f54f9754-2024-41b9-8f84-fc3d02a22253","Type":"ContainerStarted","Data":"4306113a218453fefd6fd63545507425511554f8c6b7bc99a74906a97d37cdb7"}
Oct 30 21:33:21.415970 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:21.415950    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bn8fx" event={"ID":"f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef","Type":"ContainerStarted","Data":"3b5f8b210727fb7582c0106909c9950cdf0a9bf1eb67d2a632d3ad5903943052"}
Oct 30 21:33:21.601258 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:21.600931    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83917e8e-50b6-4c52-b337-6f1be5a2c26a-service-ca-bundle\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:21.601258 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:21.600981    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-metrics-certs\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:21.601258 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:21.601017    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2qz48\" (UID: \"9f16bd1f-a3bd-45ee-a02e-cb6b9e982327\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48"
Oct 30 21:33:21.601258 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:21.601063    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d5f25065-34c9-4724-8dff-9ace0890d565-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qx7zq\" (UID: \"d5f25065-34c9-4724-8dff-9ace0890d565\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qx7zq"
Oct 30 21:33:21.601258 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:21.601085    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/83917e8e-50b6-4c52-b337-6f1be5a2c26a-service-ca-bundle podName:83917e8e-50b6-4c52-b337-6f1be5a2c26a nodeName:}" failed. No retries permitted until 2026-10-30 21:33:29.601066904 +0000 UTC m=+46.928033119 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/83917e8e-50b6-4c52-b337-6f1be5a2c26a-service-ca-bundle") pod "router-default-54cbfc65d5-sg48h" (UID: "83917e8e-50b6-4c52-b337-6f1be5a2c26a") : configmap references non-existent config key: service-ca.crt
Oct 30 21:33:21.601258 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:21.601143    2572 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:33:21.601258 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:21.601154    2572 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 30 21:33:21.601258 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:21.601176    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls podName:9f16bd1f-a3bd-45ee-a02e-cb6b9e982327 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:29.601168787 +0000 UTC m=+46.928134999 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2qz48" (UID: "9f16bd1f-a3bd-45ee-a02e-cb6b9e982327") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:33:21.601258 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:21.601203    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-metrics-certs podName:83917e8e-50b6-4c52-b337-6f1be5a2c26a nodeName:}" failed. No retries permitted until 2026-10-30 21:33:29.601190366 +0000 UTC m=+46.928156592 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-metrics-certs") pod "router-default-54cbfc65d5-sg48h" (UID: "83917e8e-50b6-4c52-b337-6f1be5a2c26a") : secret "router-metrics-certs-default" not found
Oct 30 21:33:21.601258 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:21.601217    2572 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 30 21:33:21.601258 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:21.601244    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d5f25065-34c9-4724-8dff-9ace0890d565-networking-console-plugin-cert podName:d5f25065-34c9-4724-8dff-9ace0890d565 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:29.601238173 +0000 UTC m=+46.928204384 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/d5f25065-34c9-4724-8dff-9ace0890d565-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-qx7zq" (UID: "d5f25065-34c9-4724-8dff-9ace0890d565") : secret "networking-console-plugin-cert" not found
Oct 30 21:33:21.647772 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:21.647716    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-z4jbc" podStartSLOduration=398.750782579 podStartE2EDuration="6m44.647695919s" totalImagesPullingTime="5.89691334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:21.646667928 +0000 UTC m=+38.973634165" watchObservedRunningTime="2026-10-30 21:33:21.647695919 +0000 UTC m=+38.974662153"
Oct 30 21:33:21.702891 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:21.702201    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kgrcd\" (UID: \"f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd"
Oct 30 21:33:21.702891 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:21.702262    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-tls\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:21.702891 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:21.702337    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-tls\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:21.702891 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:21.702587    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:33:21.702891 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:21.702604    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7fc9466475-jz9vh: secret "image-registry-tls" not found
Oct 30 21:33:21.702891 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:21.702661    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-tls podName:42dc1fb7-b6e0-40d2-bf53-20079815b12d nodeName:}" failed. No retries permitted until 2026-10-30 21:33:29.702644332 +0000 UTC m=+47.029610546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-tls") pod "image-registry-7fc9466475-jz9vh" (UID: "42dc1fb7-b6e0-40d2-bf53-20079815b12d") : secret "image-registry-tls" not found
Oct 30 21:33:21.702891 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:21.702731    2572 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:33:21.702891 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:21.702770    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-tls podName:f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf nodeName:}" failed. No retries permitted until 2026-10-30 21:33:29.70275846 +0000 UTC m=+47.029724672 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-kgrcd" (UID: "f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf") : secret "samples-operator-tls" not found
Oct 30 21:33:21.702891 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:21.702829    2572 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 30 21:33:21.702891 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:21.702839    2572 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74b44b76d6-l7p76: secret "image-registry-tls" not found
Oct 30 21:33:21.702891 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:21.702866    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-tls podName:392cc7ef-7cd2-4090-a246-0347d5af7bbf nodeName:}" failed. No retries permitted until 2026-10-30 21:33:29.70285712 +0000 UTC m=+47.029823332 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-tls") pod "image-registry-74b44b76d6-l7p76" (UID: "392cc7ef-7cd2-4090-a246-0347d5af7bbf") : secret "image-registry-tls" not found
Oct 30 21:33:21.803571 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:21.803533    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-metrics-tls\") pod \"dns-default-5kfvz\" (UID: \"2a216b8f-a750-4e1e-9ab1-ff6c89d416e5\") " pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:21.803744 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:21.803707    2572 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:33:21.803815 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:21.803778    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-metrics-tls podName:2a216b8f-a750-4e1e-9ab1-ff6c89d416e5 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:29.803758405 +0000 UTC m=+47.130724628 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-metrics-tls") pod "dns-default-5kfvz" (UID: "2a216b8f-a750-4e1e-9ab1-ff6c89d416e5") : secret "dns-default-metrics-tls" not found
Oct 30 21:33:21.901493 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:21.901416    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bn8fx" podStartSLOduration=399.006473135 podStartE2EDuration="6m44.901395745s" totalImagesPullingTime="5.89492261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:21.768633257 +0000 UTC m=+39.095599494" watchObservedRunningTime="2026-10-30 21:33:21.901395745 +0000 UTC m=+39.228361985"
Oct 30 21:33:21.902506 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:21.902480    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-g27zs"]
Oct 30 21:33:21.931423 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:21.931386    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:21.967010 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:21.966983    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-g27zs"]
Oct 30 21:33:21.973121 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:21.973095    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-mf2wv"]
Oct 30 21:33:21.984376 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:21.984243    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:33:21.984376 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:21.984290    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:33:21.984551 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:21.984487    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pczpt\""
Oct 30 21:33:21.992329 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:21.992307    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fjrtf"]
Oct 30 21:33:21.993829 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:21.993806    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mf2wv"
Oct 30 21:33:22.012889 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.012735    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a724a1c0-c643-49ff-a555-5bddec4bfacb-data-volume\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:22.012889 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.012784    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a724a1c0-c643-49ff-a555-5bddec4bfacb-crio-socket\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:22.012889 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.012810    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckxzq\" (UniqueName: \"kubernetes.io/projected/298f2a29-a322-4977-bd9f-d6b765eff5d1-kube-api-access-ckxzq\") pod \"migrator-66df5d496b-mf2wv\" (UID: \"298f2a29-a322-4977-bd9f-d6b765eff5d1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mf2wv"
Oct 30 21:33:22.012889 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.012855    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a724a1c0-c643-49ff-a555-5bddec4bfacb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:22.013201 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.013074    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a724a1c0-c643-49ff-a555-5bddec4bfacb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:22.013201 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.013137    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whb6q\" (UniqueName: \"kubernetes.io/projected/a724a1c0-c643-49ff-a555-5bddec4bfacb-kube-api-access-whb6q\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:22.014408 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.014387    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fjrtf"
Oct 30 21:33:22.035499 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.035449    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-mf2wv"]
Oct 30 21:33:22.045383 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.045336    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fjrtf"]
Oct 30 21:33:22.072265 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.072235    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-4sgw4-zcs8v"]
Oct 30 21:33:22.087845 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.087821    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4sgw4-zcs8v"
Oct 30 21:33:22.088288 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.088239    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 21:33:22.088288 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.088254    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 21:33:22.088288 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.088246    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-9nw4b\""
Oct 30 21:33:22.115396 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.114075    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a724a1c0-c643-49ff-a555-5bddec4bfacb-data-volume\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:22.115396 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.114112    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a724a1c0-c643-49ff-a555-5bddec4bfacb-crio-socket\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:22.115396 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.114136    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckxzq\" (UniqueName: \"kubernetes.io/projected/298f2a29-a322-4977-bd9f-d6b765eff5d1-kube-api-access-ckxzq\") pod \"migrator-66df5d496b-mf2wv\" (UID: \"298f2a29-a322-4977-bd9f-d6b765eff5d1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mf2wv"
Oct 30 21:33:22.115396 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.114168    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5278b436-9321-4856-b9b0-ce9eb2596775-service-ca-bundle\") pod \"periodic-gathering-4sgw4-zcs8v\" (UID: \"5278b436-9321-4856-b9b0-ce9eb2596775\") " pod="openshift-insights/periodic-gathering-4sgw4-zcs8v"
Oct 30 21:33:22.115396 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.114210    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a724a1c0-c643-49ff-a555-5bddec4bfacb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:22.115396 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.114259    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5278b436-9321-4856-b9b0-ce9eb2596775-trusted-ca-bundle\") pod \"periodic-gathering-4sgw4-zcs8v\" (UID: \"5278b436-9321-4856-b9b0-ce9eb2596775\") " pod="openshift-insights/periodic-gathering-4sgw4-zcs8v"
Oct 30 21:33:22.115396 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.114327    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6gq8\" (UniqueName: \"kubernetes.io/projected/3d0811a7-4a83-4758-8ca0-2ac7dd5a9408-kube-api-access-h6gq8\") pod \"migrator-66df5d496b-fjrtf\" (UID: \"3d0811a7-4a83-4758-8ca0-2ac7dd5a9408\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fjrtf"
Oct 30 21:33:22.115396 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.114407    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a724a1c0-c643-49ff-a555-5bddec4bfacb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:22.115396 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.114462    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whb6q\" (UniqueName: \"kubernetes.io/projected/a724a1c0-c643-49ff-a555-5bddec4bfacb-kube-api-access-whb6q\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:22.115396 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.114487    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5278b436-9321-4856-b9b0-ce9eb2596775-archives-path\") pod \"periodic-gathering-4sgw4-zcs8v\" (UID: \"5278b436-9321-4856-b9b0-ce9eb2596775\") " pod="openshift-insights/periodic-gathering-4sgw4-zcs8v"
Oct 30 21:33:22.115396 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.114512    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48g5f\" (UniqueName: \"kubernetes.io/projected/5278b436-9321-4856-b9b0-ce9eb2596775-kube-api-access-48g5f\") pod \"periodic-gathering-4sgw4-zcs8v\" (UID: \"5278b436-9321-4856-b9b0-ce9eb2596775\") " pod="openshift-insights/periodic-gathering-4sgw4-zcs8v"
Oct 30 21:33:22.115396 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.114782    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a724a1c0-c643-49ff-a555-5bddec4bfacb-crio-socket\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:22.115396 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.114850    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a724a1c0-c643-49ff-a555-5bddec4bfacb-data-volume\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:22.115396 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:22.114952    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:33:22.115396 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:22.115003    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a724a1c0-c643-49ff-a555-5bddec4bfacb-kube-rbac-proxy-cm podName:a724a1c0-c643-49ff-a555-5bddec4bfacb nodeName:}" failed. No retries permitted until 2026-10-30 21:33:22.614986621 +0000 UTC m=+39.941952836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a724a1c0-c643-49ff-a555-5bddec4bfacb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g27zs" (UID: "a724a1c0-c643-49ff-a555-5bddec4bfacb") : configmap "kube-rbac-proxy" not found
Oct 30 21:33:22.115396 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:22.115298    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:33:22.116173 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:22.115371    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a724a1c0-c643-49ff-a555-5bddec4bfacb-insights-runtime-extractor-tls podName:a724a1c0-c643-49ff-a555-5bddec4bfacb nodeName:}" failed. No retries permitted until 2026-10-30 21:33:22.61533924 +0000 UTC m=+39.942305457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a724a1c0-c643-49ff-a555-5bddec4bfacb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g27zs" (UID: "a724a1c0-c643-49ff-a555-5bddec4bfacb") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:33:22.132840 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.132815    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-4sgw4-zcs8v"]
Oct 30 21:33:22.173142 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.173112    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whb6q\" (UniqueName: \"kubernetes.io/projected/a724a1c0-c643-49ff-a555-5bddec4bfacb-kube-api-access-whb6q\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:22.173142 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.173135    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckxzq\" (UniqueName: \"kubernetes.io/projected/298f2a29-a322-4977-bd9f-d6b765eff5d1-kube-api-access-ckxzq\") pod \"migrator-66df5d496b-mf2wv\" (UID: \"298f2a29-a322-4977-bd9f-d6b765eff5d1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mf2wv"
Oct 30 21:33:22.216039 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.215964    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6gq8\" (UniqueName: \"kubernetes.io/projected/3d0811a7-4a83-4758-8ca0-2ac7dd5a9408-kube-api-access-h6gq8\") pod \"migrator-66df5d496b-fjrtf\" (UID: \"3d0811a7-4a83-4758-8ca0-2ac7dd5a9408\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fjrtf"
Oct 30 21:33:22.216175 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.216046    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5278b436-9321-4856-b9b0-ce9eb2596775-archives-path\") pod \"periodic-gathering-4sgw4-zcs8v\" (UID: \"5278b436-9321-4856-b9b0-ce9eb2596775\") " pod="openshift-insights/periodic-gathering-4sgw4-zcs8v"
Oct 30 21:33:22.216175 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.216072    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48g5f\" (UniqueName: \"kubernetes.io/projected/5278b436-9321-4856-b9b0-ce9eb2596775-kube-api-access-48g5f\") pod \"periodic-gathering-4sgw4-zcs8v\" (UID: \"5278b436-9321-4856-b9b0-ce9eb2596775\") " pod="openshift-insights/periodic-gathering-4sgw4-zcs8v"
Oct 30 21:33:22.216175 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.216142    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5278b436-9321-4856-b9b0-ce9eb2596775-service-ca-bundle\") pod \"periodic-gathering-4sgw4-zcs8v\" (UID: \"5278b436-9321-4856-b9b0-ce9eb2596775\") " pod="openshift-insights/periodic-gathering-4sgw4-zcs8v"
Oct 30 21:33:22.216631 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.216433    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5278b436-9321-4856-b9b0-ce9eb2596775-trusted-ca-bundle\") pod \"periodic-gathering-4sgw4-zcs8v\" (UID: \"5278b436-9321-4856-b9b0-ce9eb2596775\") " pod="openshift-insights/periodic-gathering-4sgw4-zcs8v"
Oct 30 21:33:22.216631 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.216497    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5278b436-9321-4856-b9b0-ce9eb2596775-archives-path\") pod \"periodic-gathering-4sgw4-zcs8v\" (UID: \"5278b436-9321-4856-b9b0-ce9eb2596775\") " pod="openshift-insights/periodic-gathering-4sgw4-zcs8v"
Oct 30 21:33:22.216819 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.216803    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5278b436-9321-4856-b9b0-ce9eb2596775-service-ca-bundle\") pod \"periodic-gathering-4sgw4-zcs8v\" (UID: \"5278b436-9321-4856-b9b0-ce9eb2596775\") " pod="openshift-insights/periodic-gathering-4sgw4-zcs8v"
Oct 30 21:33:22.217278 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.217251    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5278b436-9321-4856-b9b0-ce9eb2596775-trusted-ca-bundle\") pod \"periodic-gathering-4sgw4-zcs8v\" (UID: \"5278b436-9321-4856-b9b0-ce9eb2596775\") " pod="openshift-insights/periodic-gathering-4sgw4-zcs8v"
Oct 30 21:33:22.273964 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.273901    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48g5f\" (UniqueName: \"kubernetes.io/projected/5278b436-9321-4856-b9b0-ce9eb2596775-kube-api-access-48g5f\") pod \"periodic-gathering-4sgw4-zcs8v\" (UID: \"5278b436-9321-4856-b9b0-ce9eb2596775\") " pod="openshift-insights/periodic-gathering-4sgw4-zcs8v"
Oct 30 21:33:22.275872 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.275819    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6gq8\" (UniqueName: \"kubernetes.io/projected/3d0811a7-4a83-4758-8ca0-2ac7dd5a9408-kube-api-access-h6gq8\") pod \"migrator-66df5d496b-fjrtf\" (UID: \"3d0811a7-4a83-4758-8ca0-2ac7dd5a9408\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fjrtf"
Oct 30 21:33:22.349093 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.349063    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mf2wv"
Oct 30 21:33:22.364634 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.364606    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fjrtf"
Oct 30 21:33:22.415830 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.415551    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4sgw4-zcs8v"
Oct 30 21:33:22.429216 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.429142    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ljjfm" event={"ID":"b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5","Type":"ContainerStarted","Data":"557dcc42b46ca3d7394fdf94bc0ad0abdf303f1821f95b3bf6c044519f6ee356"}
Oct 30 21:33:22.432125 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.431482    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 21:33:22.432125 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.431871    2572 scope.go:122] "RemoveContainer" containerID="d036f501bf39148b5ef1dfc318a42d74c69b78f8ea89630deb12c0797ede2521"
Oct 30 21:33:22.432125 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:22.432055    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-z8clf_openshift-console-operator(1196dab1-80d7-48eb-8d05-1a26ab6d0759)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf" podUID="1196dab1-80d7-48eb-8d05-1a26ab6d0759"
Oct 30 21:33:22.434377 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.434277    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-64b9m" event={"ID":"579a0c8f-9497-4448-98ab-4686adec22cd","Type":"ContainerStarted","Data":"90031a864dfd6a44f6ac62fce7bb7b240f0d0cc24192ba579f20797e0f3db470"}
Oct 30 21:33:22.543705 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.543648    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ljjfm" podStartSLOduration=2.708969435 podStartE2EDuration="9.543630661s" totalImagesPullingTime="6.834661226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:22.543282039 +0000 UTC m=+39.870248271" watchObservedRunningTime="2026-10-30 21:33:22.543630661 +0000 UTC m=+39.870596895"
Oct 30 21:33:22.621423 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.620716    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a724a1c0-c643-49ff-a555-5bddec4bfacb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:22.621423 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.620833    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a724a1c0-c643-49ff-a555-5bddec4bfacb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:22.621423 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:22.620955    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:33:22.621423 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:22.621008    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a724a1c0-c643-49ff-a555-5bddec4bfacb-kube-rbac-proxy-cm podName:a724a1c0-c643-49ff-a555-5bddec4bfacb nodeName:}" failed. No retries permitted until 2026-10-30 21:33:23.620994993 +0000 UTC m=+40.947961218 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a724a1c0-c643-49ff-a555-5bddec4bfacb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g27zs" (UID: "a724a1c0-c643-49ff-a555-5bddec4bfacb") : configmap "kube-rbac-proxy" not found
Oct 30 21:33:22.621423 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:22.621085    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:33:22.621423 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:22.621110    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a724a1c0-c643-49ff-a555-5bddec4bfacb-insights-runtime-extractor-tls podName:a724a1c0-c643-49ff-a555-5bddec4bfacb nodeName:}" failed. No retries permitted until 2026-10-30 21:33:23.621103006 +0000 UTC m=+40.948069231 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a724a1c0-c643-49ff-a555-5bddec4bfacb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g27zs" (UID: "a724a1c0-c643-49ff-a555-5bddec4bfacb") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:33:22.658336 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.658284    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-64b9m" podStartSLOduration=398.481471441 podStartE2EDuration="6m45.658268131s" totalImagesPullingTime="7.17679669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:22.656404988 +0000 UTC m=+39.983371220" watchObservedRunningTime="2026-10-30 21:33:22.658268131 +0000 UTC m=+39.985234371"
Oct 30 21:33:22.734776 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.734741    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-mf2wv"]
Oct 30 21:33:22.736739 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:33:22.736701    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod298f2a29_a322_4977_bd9f_d6b765eff5d1.slice/crio-31c6d1edb618126ab90cd9d1ced9d89feb0db514777f385fce55ca2540662aa0 WatchSource:0}: Error finding container 31c6d1edb618126ab90cd9d1ced9d89feb0db514777f385fce55ca2540662aa0: Status 404 returned error can't find the container with id 31c6d1edb618126ab90cd9d1ced9d89feb0db514777f385fce55ca2540662aa0
Oct 30 21:33:22.784700 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.784618    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-fjrtf"]
Oct 30 21:33:22.789289 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:33:22.789260    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d0811a7_4a83_4758_8ca0_2ac7dd5a9408.slice/crio-76a2578487fd0958df5ea2687eadc05751d24772531eb6c223fcb41eecec2f52 WatchSource:0}: Error finding container 76a2578487fd0958df5ea2687eadc05751d24772531eb6c223fcb41eecec2f52: Status 404 returned error can't find the container with id 76a2578487fd0958df5ea2687eadc05751d24772531eb6c223fcb41eecec2f52
Oct 30 21:33:22.828564 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:22.828248    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-4sgw4-zcs8v"]
Oct 30 21:33:22.830381 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:33:22.830330    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5278b436_9321_4856_b9b0_ce9eb2596775.slice/crio-1e778827f2aa19da409b22ecbb0cc7a80f0e711817a34162237798e464a6c01e WatchSource:0}: Error finding container 1e778827f2aa19da409b22ecbb0cc7a80f0e711817a34162237798e464a6c01e: Status 404 returned error can't find the container with id 1e778827f2aa19da409b22ecbb0cc7a80f0e711817a34162237798e464a6c01e
Oct 30 21:33:23.085648 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:23.085616    2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4cfcr"]
Oct 30 21:33:23.124058 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:23.124035    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-4cfcr"
Oct 30 21:33:23.147678 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:23.147619    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4cfcr"]
Oct 30 21:33:23.176812 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:23.176784    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-cftmq\""
Oct 30 21:33:23.176812 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:23.176795    2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 21:33:23.206072 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:23.206045    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 21:33:23.206279 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:23.206100    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 21:33:23.206279 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:23.206134    2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 21:33:23.227292 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:23.227263    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/98a3aaba-8c58-43ae-837d-a809085d692c-signing-cabundle\") pod \"service-ca-5fffc97bdf-4cfcr\" (UID: \"98a3aaba-8c58-43ae-837d-a809085d692c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4cfcr"
Oct 30 21:33:23.227458 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:23.227359    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fbj2\" (UniqueName: \"kubernetes.io/projected/98a3aaba-8c58-43ae-837d-a809085d692c-kube-api-access-4fbj2\") pod \"service-ca-5fffc97bdf-4cfcr\" (UID: \"98a3aaba-8c58-43ae-837d-a809085d692c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4cfcr"
Oct 30 21:33:23.227608 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:23.227583    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/98a3aaba-8c58-43ae-837d-a809085d692c-signing-key\") pod \"service-ca-5fffc97bdf-4cfcr\" (UID: \"98a3aaba-8c58-43ae-837d-a809085d692c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4cfcr"
Oct 30 21:33:23.328378 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:23.328273    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/98a3aaba-8c58-43ae-837d-a809085d692c-signing-cabundle\") pod \"service-ca-5fffc97bdf-4cfcr\" (UID: \"98a3aaba-8c58-43ae-837d-a809085d692c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4cfcr"
Oct 30 21:33:23.328555 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:23.328385    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fbj2\" (UniqueName: \"kubernetes.io/projected/98a3aaba-8c58-43ae-837d-a809085d692c-kube-api-access-4fbj2\") pod \"service-ca-5fffc97bdf-4cfcr\" (UID: \"98a3aaba-8c58-43ae-837d-a809085d692c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4cfcr"
Oct 30 21:33:23.328625 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:23.328559    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/98a3aaba-8c58-43ae-837d-a809085d692c-signing-key\") pod \"service-ca-5fffc97bdf-4cfcr\" (UID: \"98a3aaba-8c58-43ae-837d-a809085d692c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4cfcr"
Oct 30 21:33:23.329174 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:23.329142    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/98a3aaba-8c58-43ae-837d-a809085d692c-signing-cabundle\") pod \"service-ca-5fffc97bdf-4cfcr\" (UID: \"98a3aaba-8c58-43ae-837d-a809085d692c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4cfcr"
Oct 30 21:33:23.331281 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:23.331259    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/98a3aaba-8c58-43ae-837d-a809085d692c-signing-key\") pod \"service-ca-5fffc97bdf-4cfcr\" (UID: \"98a3aaba-8c58-43ae-837d-a809085d692c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4cfcr"
Oct 30 21:33:23.385937 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:23.385908    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fbj2\" (UniqueName: \"kubernetes.io/projected/98a3aaba-8c58-43ae-837d-a809085d692c-kube-api-access-4fbj2\") pod \"service-ca-5fffc97bdf-4cfcr\" (UID: \"98a3aaba-8c58-43ae-837d-a809085d692c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4cfcr"
Oct 30 21:33:23.441013 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:23.440979    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-spx9v" event={"ID":"e1a448f1-e180-4fbf-9923-fac0b9dbec74","Type":"ContainerStarted","Data":"e0e4bde082e2f5105bff0b28c5099edb09eb72d3465535765ee172322fb69398"}
Oct 30 21:33:23.441501 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:23.441173    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:33:23.442392 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:23.442338    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fjrtf" event={"ID":"3d0811a7-4a83-4758-8ca0-2ac7dd5a9408","Type":"ContainerStarted","Data":"76a2578487fd0958df5ea2687eadc05751d24772531eb6c223fcb41eecec2f52"}
Oct 30 21:33:23.443751 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:23.443724    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mf2wv" event={"ID":"298f2a29-a322-4977-bd9f-d6b765eff5d1","Type":"ContainerStarted","Data":"31c6d1edb618126ab90cd9d1ced9d89feb0db514777f385fce55ca2540662aa0"}
Oct 30 21:33:23.445532 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:23.445235    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4sgw4-zcs8v" event={"ID":"5278b436-9321-4856-b9b0-ce9eb2596775","Type":"ContainerStarted","Data":"6c85c9004b2949eb998a21c5af46307116a0c21a749672fc775ee8df1cc0f88a"}
Oct 30 21:33:23.445532 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:23.445283    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4sgw4-zcs8v" event={"ID":"5278b436-9321-4856-b9b0-ce9eb2596775","Type":"ContainerStarted","Data":"1e778827f2aa19da409b22ecbb0cc7a80f0e711817a34162237798e464a6c01e"}
Oct 30 21:33:23.446680 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:23.446631    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-4cfcr"
Oct 30 21:33:23.558898 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:23.558850    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-spx9v" podStartSLOduration=34.34840373 podStartE2EDuration="40.558834664s" totalImagesPullingTime="6.210430934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:32:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:23.557551026 +0000 UTC m=+40.884517260" watchObservedRunningTime="2026-10-30 21:33:23.558834664 +0000 UTC m=+40.885800911"
Oct 30 21:33:23.632159 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:23.632083    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a724a1c0-c643-49ff-a555-5bddec4bfacb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:23.632313 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:23.632245    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a724a1c0-c643-49ff-a555-5bddec4bfacb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:23.632313 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:23.632256    2572 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:33:23.632442 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:23.632337    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a724a1c0-c643-49ff-a555-5bddec4bfacb-insights-runtime-extractor-tls podName:a724a1c0-c643-49ff-a555-5bddec4bfacb nodeName:}" failed. No retries permitted until 2026-10-30 21:33:25.632313925 +0000 UTC m=+42.959280156 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a724a1c0-c643-49ff-a555-5bddec4bfacb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g27zs" (UID: "a724a1c0-c643-49ff-a555-5bddec4bfacb") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:33:23.632442 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:23.632379    2572 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:33:23.632442 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:23.632434    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a724a1c0-c643-49ff-a555-5bddec4bfacb-kube-rbac-proxy-cm podName:a724a1c0-c643-49ff-a555-5bddec4bfacb nodeName:}" failed. No retries permitted until 2026-10-30 21:33:25.632418084 +0000 UTC m=+42.959384298 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a724a1c0-c643-49ff-a555-5bddec4bfacb-kube-rbac-proxy-cm") pod "insights-runtime-extractor-g27zs" (UID: "a724a1c0-c643-49ff-a555-5bddec4bfacb") : configmap "kube-rbac-proxy" not found
Oct 30 21:33:23.676497 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:23.676443    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-4sgw4-zcs8v" podStartSLOduration=1.676428456 podStartE2EDuration="1.676428456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:23.674777206 +0000 UTC m=+41.001743443" watchObservedRunningTime="2026-10-30 21:33:23.676428456 +0000 UTC m=+41.003394689"
Oct 30 21:33:23.850580 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:23.849837    2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4cfcr"]
Oct 30 21:33:23.858790 ip-10-0-132-92 kubenswrapper[2572]: W1030 21:33:23.858757    2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a3aaba_8c58_43ae_837d_a809085d692c.slice/crio-c3b73763545b02c62768710b9a3080c79703fc94c6cdbc27870033e6d52ec19c WatchSource:0}: Error finding container c3b73763545b02c62768710b9a3080c79703fc94c6cdbc27870033e6d52ec19c: Status 404 returned error can't find the container with id c3b73763545b02c62768710b9a3080c79703fc94c6cdbc27870033e6d52ec19c
Oct 30 21:33:24.452607 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:24.452580    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-4cfcr" event={"ID":"98a3aaba-8c58-43ae-837d-a809085d692c","Type":"ContainerStarted","Data":"746730343d41333737e6e4736e0dab00e934b27a5705e0018d011be64d66095b"}
Oct 30 21:33:24.453062 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:24.452621    2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-4cfcr" event={"ID":"98a3aaba-8c58-43ae-837d-a809085d692c","Type":"ContainerStarted","Data":"c3b73763545b02c62768710b9a3080c79703fc94c6cdbc27870033e6d52ec19c"}
Oct 30 21:33:24.468962 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:24.468934    2572 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf"
Oct 30 21:33:24.469316 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:24.469293    2572 scope.go:122] "RemoveContainer" containerID="d036f501bf39148b5ef1dfc318a42d74c69b78f8ea89630deb12c0797ede2521"
Oct 30 21:33:24.469487 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:24.469467    2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf"
Oct 30 21:33:24.469553 ip-10-0-132-92 kubenswrapper[2572]: E1030 21:33:24.469514    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-z8clf_openshift-console-operator(1196dab1-80d7-48eb-8d05-1a26ab6d0759)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf" podUID="1196dab1-80d7-48eb-8d05-1a26ab6d0759"
Oct 30 21:33:24.565549 ip-10-0-132-92 kubenswrapper[2572]: I1030 21:33:24.565499    2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-4cfcr" podStartSLOduration=1.565483918 podStartE2EDuration="1.565483918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:24.56486027 +0000 UTC m=+41.891826506" watchObservedRunningTime="2026-10-30 21:33:24.565483918 +0000 UTC m=+41.892450186"
Oct 30 21:33:25.442117 ip-10-0-132-92 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 21:33:25.448045 ip-10-0-132-92 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 21:33:25.448217 ip-10-0-132-92 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 21:33:25.448269 ip-10-0-132-92 systemd[1]: kubelet.service: Consumed 3.200s CPU time.
Oct 30 21:33:25.468806 ip-10-0-132-92 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:33:25.586907 ip-10-0-132-92 kubenswrapper[5087]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:33:25.586907 ip-10-0-132-92 kubenswrapper[5087]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:33:25.586907 ip-10-0-132-92 kubenswrapper[5087]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:33:25.586907 ip-10-0-132-92 kubenswrapper[5087]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:33:25.590231 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590131    5087 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:33:25.590231 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590233    5087 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:33:25.590334 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590239    5087 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:33:25.590334 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590243    5087 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:33:25.590334 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590248    5087 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:33:25.590334 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590252    5087 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:33:25.590334 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590257    5087 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:33:25.590334 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590267    5087 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:33:25.590334 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590272    5087 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:33:25.590334 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590276    5087 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:33:25.590334 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590280    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:33:25.590334 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590285    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:33:25.590334 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590291    5087 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:33:25.590334 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590297    5087 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:33:25.590334 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590304    5087 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:33:25.590334 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590310    5087 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:33:25.590334 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590315    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:33:25.590334 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590321    5087 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:33:25.590334 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590326    5087 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:33:25.590334 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590331    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:33:25.591078 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590336    5087 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:33:25.591078 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590341    5087 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:33:25.591078 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590362    5087 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:33:25.591078 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590367    5087 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:33:25.591078 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590371    5087 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:33:25.591078 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590376    5087 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:33:25.591078 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590380    5087 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:33:25.591078 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590384    5087 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:33:25.591078 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590388    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:33:25.591078 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590392    5087 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:33:25.591078 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590396    5087 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:33:25.591078 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590402    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:33:25.591078 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590406    5087 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:33:25.591078 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590410    5087 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:33:25.591078 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590414    5087 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:33:25.591078 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590418    5087 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:33:25.591078 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590422    5087 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:33:25.591078 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590426    5087 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:33:25.591078 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590430    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:33:25.591078 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590435    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:33:25.591921 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590439    5087 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:33:25.591921 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590443    5087 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:33:25.591921 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590447    5087 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:33:25.591921 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590451    5087 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:33:25.591921 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590455    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:33:25.591921 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590459    5087 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:33:25.591921 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590463    5087 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:33:25.591921 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590467    5087 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:33:25.591921 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590471    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:33:25.591921 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590475    5087 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:33:25.591921 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590479    5087 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:33:25.591921 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590483    5087 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:33:25.591921 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590487    5087 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:33:25.591921 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590492    5087 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:33:25.591921 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590496    5087 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:33:25.591921 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590500    5087 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:33:25.591921 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590504    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:33:25.591921 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590508    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:33:25.591921 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590512    5087 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:33:25.592785 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590516    5087 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:33:25.592785 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590520    5087 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:33:25.592785 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590524    5087 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:33:25.592785 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590528    5087 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:33:25.592785 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590533    5087 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:33:25.592785 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590537    5087 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:33:25.592785 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590541    5087 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:33:25.592785 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590545    5087 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:33:25.592785 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590549    5087 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:33:25.592785 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590553    5087 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:33:25.592785 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590557    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:33:25.592785 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590561    5087 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:33:25.592785 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590565    5087 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:33:25.592785 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590570    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:33:25.592785 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590574    5087 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:33:25.592785 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590578    5087 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:33:25.592785 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590582    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:33:25.592785 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590586    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:33:25.592785 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590590    5087 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:33:25.592785 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590595    5087 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:33:25.593646 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590599    5087 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:33:25.593646 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590603    5087 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:33:25.593646 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590607    5087 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:33:25.593646 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590611    5087 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:33:25.593646 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590615    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:33:25.593646 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590619    5087 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:33:25.593646 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590622    5087 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:33:25.593646 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590626    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:33:25.593646 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590630    5087 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:33:25.593646 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590634    5087 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:33:25.593646 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590638    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:33:25.593646 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590642    5087 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:33:25.593646 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590646    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:33:25.593646 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590650    5087 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:33:25.593646 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590654    5087 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:33:25.593646 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590658    5087 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:33:25.593646 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590662    5087 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:33:25.593646 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590666    5087 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:33:25.593646 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590670    5087 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:33:25.593646 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590674    5087 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:33:25.594486 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590678    5087 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:33:25.594486 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590682    5087 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:33:25.594486 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590686    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:33:25.594486 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590690    5087 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:33:25.594486 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590695    5087 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:33:25.594486 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590700    5087 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:33:25.594486 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.590705    5087 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:33:25.594486 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592223    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:33:25.594486 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592233    5087 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:33:25.594486 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592238    5087 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:33:25.594486 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592242    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:33:25.594486 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592246    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:33:25.594486 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592251    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:33:25.594486 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592255    5087 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:33:25.594486 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592259    5087 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:33:25.594486 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592263    5087 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:33:25.594486 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592267    5087 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:33:25.594486 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592271    5087 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:33:25.594486 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592276    5087 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:33:25.595250 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592280    5087 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:33:25.595250 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592284    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:33:25.595250 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592288    5087 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:33:25.595250 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592292    5087 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:33:25.595250 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592297    5087 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:33:25.595250 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592301    5087 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:33:25.595250 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592305    5087 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:33:25.595250 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592309    5087 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:33:25.595250 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592313    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:33:25.595250 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592317    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:33:25.595250 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592321    5087 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:33:25.595250 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592325    5087 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:33:25.595250 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592330    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:33:25.595250 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592335    5087 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:33:25.595250 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592339    5087 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:33:25.595250 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592367    5087 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:33:25.595250 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592372    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:33:25.595250 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592376    5087 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:33:25.595250 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592380    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:33:25.595250 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592385    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:33:25.596117 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592402    5087 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:33:25.596117 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592406    5087 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:33:25.596117 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592411    5087 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:33:25.596117 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592415    5087 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:33:25.596117 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592419    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:33:25.596117 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592423    5087 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:33:25.596117 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592427    5087 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:33:25.596117 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592431    5087 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:33:25.596117 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592436    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:33:25.596117 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592440    5087 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:33:25.596117 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592444    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:33:25.596117 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592448    5087 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:33:25.596117 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592452    5087 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:33:25.596117 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592456    5087 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:33:25.596117 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592460    5087 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:33:25.596117 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592464    5087 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:33:25.596117 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592468    5087 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:33:25.596117 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592472    5087 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:33:25.596117 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592477    5087 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:33:25.596900 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592481    5087 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:33:25.596900 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592488    5087 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:33:25.596900 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592494    5087 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:33:25.596900 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592498    5087 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:33:25.596900 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592503    5087 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:33:25.596900 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592507    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:33:25.596900 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592511    5087 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:33:25.596900 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592516    5087 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:33:25.596900 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592520    5087 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:33:25.596900 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592524    5087 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:33:25.596900 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592528    5087 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:33:25.596900 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592532    5087 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:33:25.596900 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592536    5087 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:33:25.596900 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592540    5087 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:33:25.596900 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592545    5087 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:33:25.596900 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592549    5087 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:33:25.596900 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592553    5087 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:33:25.596900 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592557    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:33:25.596900 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592561    5087 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:33:25.596900 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592565    5087 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:33:25.597739 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592569    5087 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:33:25.597739 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592573    5087 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:33:25.597739 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592577    5087 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:33:25.597739 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592581    5087 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:33:25.597739 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592585    5087 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:33:25.597739 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592589    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:33:25.597739 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592593    5087 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:33:25.597739 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592600    5087 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:33:25.597739 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592605    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:33:25.597739 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592610    5087 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:33:25.597739 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592616    5087 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:33:25.597739 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592620    5087 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:33:25.597739 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592624    5087 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:33:25.597739 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592629    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:33:25.597739 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592634    5087 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:33:25.597739 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592638    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:33:25.597739 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592642    5087 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:33:25.597739 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592647    5087 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:33:25.597739 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592651    5087 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:33:25.598554 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592655    5087 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:33:25.598554 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592659    5087 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:33:25.598554 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592664    5087 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:33:25.598554 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592668    5087 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:33:25.598554 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592672    5087 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:33:25.598554 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592677    5087 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:33:25.598554 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592681    5087 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:33:25.598554 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592685    5087 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:33:25.598554 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592690    5087 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:33:25.598554 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592694    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:33:25.598554 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592698    5087 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:33:25.598554 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592702    5087 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:33:25.598554 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592707    5087 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:33:25.598554 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592711    5087 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:33:25.598554 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592715    5087 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:33:25.598554 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.592721    5087 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:33:25.598554 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593476    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:33:25.598554 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593483    5087 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:33:25.598554 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593492    5087 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:33:25.598554 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593496    5087 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:33:25.599359 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593500    5087 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:33:25.599359 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593504    5087 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:33:25.599359 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593508    5087 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:33:25.599359 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593513    5087 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:33:25.599359 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593517    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:33:25.599359 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593521    5087 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:33:25.599359 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593525    5087 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:33:25.599359 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593529    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:33:25.599359 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593533    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:33:25.599359 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593537    5087 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:33:25.599359 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593541    5087 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:33:25.599359 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593545    5087 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:33:25.599359 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593549    5087 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:33:25.599359 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593552    5087 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:33:25.599359 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593556    5087 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:33:25.599359 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593560    5087 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:33:25.599359 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593564    5087 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:33:25.599359 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593568    5087 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:33:25.599359 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593573    5087 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:33:25.599359 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593577    5087 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:33:25.600172 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593582    5087 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:33:25.600172 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593586    5087 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:33:25.600172 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593590    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:33:25.600172 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593594    5087 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:33:25.600172 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593600    5087 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:33:25.600172 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593606    5087 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:33:25.600172 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593612    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:33:25.600172 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593618    5087 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:33:25.600172 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593622    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:33:25.600172 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593627    5087 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:33:25.600172 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593632    5087 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:33:25.600172 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593636    5087 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:33:25.600172 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593640    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:33:25.600172 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593644    5087 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:33:25.600172 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593648    5087 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:33:25.600172 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593653    5087 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:33:25.600172 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593657    5087 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:33:25.600172 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593663    5087 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:33:25.600172 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593667    5087 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:33:25.600985 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593671    5087 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:33:25.600985 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593675    5087 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:33:25.600985 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593679    5087 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:33:25.600985 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593684    5087 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:33:25.600985 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593688    5087 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:33:25.600985 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593692    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:33:25.600985 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593696    5087 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:33:25.600985 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593700    5087 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:33:25.600985 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593704    5087 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:33:25.600985 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593707    5087 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:33:25.600985 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593711    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:33:25.600985 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593715    5087 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:33:25.600985 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593719    5087 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:33:25.600985 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593723    5087 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:33:25.600985 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593732    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:33:25.600985 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593736    5087 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:33:25.600985 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593741    5087 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:33:25.600985 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593744    5087 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:33:25.600985 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593748    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:33:25.600985 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593752    5087 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:33:25.601802 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593756    5087 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:33:25.601802 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593760    5087 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:33:25.601802 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593764    5087 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:33:25.601802 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593770    5087 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:33:25.601802 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593774    5087 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:33:25.601802 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593778    5087 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:33:25.601802 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593781    5087 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:33:25.601802 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593785    5087 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:33:25.601802 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593789    5087 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:33:25.601802 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593793    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:33:25.601802 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593798    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:33:25.601802 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593803    5087 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:33:25.601802 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593807    5087 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:33:25.601802 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593811    5087 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:33:25.601802 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593814    5087 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:33:25.601802 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593818    5087 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:33:25.601802 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593822    5087 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:33:25.601802 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593826    5087 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:33:25.601802 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593831    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:33:25.601802 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593836    5087 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:33:25.602613 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593840    5087 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:33:25.602613 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593844    5087 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:33:25.602613 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593848    5087 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:33:25.602613 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593852    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:33:25.602613 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593856    5087 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:33:25.602613 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593860    5087 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:33:25.602613 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593864    5087 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:33:25.602613 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593869    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:33:25.602613 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593873    5087 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:33:25.602613 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593877    5087 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:33:25.602613 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593881    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:33:25.602613 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593884    5087 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:33:25.602613 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593888    5087 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:33:25.602613 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593893    5087 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:33:25.602613 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593896    5087 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:33:25.602613 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593902    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:33:25.602613 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593906    5087 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:33:25.602613 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593910    5087 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:33:25.602613 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593914    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:33:25.603397 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593918    5087 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:33:25.603397 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593921    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:33:25.603397 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593925    5087 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:33:25.603397 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.593931    5087 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:33:25.603397 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594102    5087 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:33:25.603397 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594113    5087 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:33:25.603397 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594123    5087 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:33:25.603397 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594129    5087 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:33:25.603397 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594136    5087 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:33:25.603397 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594141    5087 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:33:25.603397 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594147    5087 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:33:25.603397 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594153    5087 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:33:25.603397 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594158    5087 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:33:25.603397 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594163    5087 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:33:25.603397 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594168    5087 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:33:25.603397 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594173    5087 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:33:25.603397 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594178    5087 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:33:25.603397 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594183    5087 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:33:25.603397 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594187    5087 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:33:25.603397 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594192    5087 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:33:25.603397 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594196    5087 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:33:25.603397 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594201    5087 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:33:25.603397 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594208    5087 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:33:25.604307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594212    5087 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:33:25.604307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594217    5087 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:33:25.604307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594222    5087 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:33:25.604307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594227    5087 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:33:25.604307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594233    5087 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:33:25.604307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594240    5087 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:33:25.604307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594245    5087 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:33:25.604307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594250    5087 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:33:25.604307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594254    5087 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:33:25.604307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594259    5087 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:33:25.604307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594264    5087 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:33:25.604307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594269    5087 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:33:25.604307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594273    5087 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:33:25.604307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594280    5087 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:33:25.604307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594285    5087 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:33:25.604307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594289    5087 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:33:25.604307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594294    5087 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:33:25.604307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594298    5087 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:33:25.604307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594303    5087 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:33:25.604307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594309    5087 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:33:25.604307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594314    5087 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:33:25.604307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594319    5087 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:33:25.604307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594323    5087 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:33:25.604307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594328    5087 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:33:25.605407 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594334    5087 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:33:25.605407 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594339    5087 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:33:25.605407 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594358    5087 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:33:25.605407 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594363    5087 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:33:25.605407 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594368    5087 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:33:25.605407 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594372    5087 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:33:25.605407 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594376    5087 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:33:25.605407 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594382    5087 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:33:25.605407 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594386    5087 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:33:25.605407 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594391    5087 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:33:25.605407 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594395    5087 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:33:25.605407 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594401    5087 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:33:25.605407 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594406    5087 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:33:25.605407 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594412    5087 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:33:25.605407 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594417    5087 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:33:25.605407 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594422    5087 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:33:25.605407 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594426    5087 flags.go:64] FLAG: --help="false"
Oct 30 21:33:25.605407 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594431    5087 flags.go:64] FLAG: --hostname-override="ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:33:25.605407 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594436    5087 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:33:25.605407 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594440    5087 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:33:25.605407 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594445    5087 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:33:25.605407 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594450    5087 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:33:25.605407 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594456    5087 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:33:25.606382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594460    5087 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:33:25.606382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594464    5087 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:33:25.606382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594469    5087 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:33:25.606382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594473    5087 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:33:25.606382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594478    5087 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:33:25.606382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594482    5087 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:33:25.606382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594487    5087 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:33:25.606382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594492    5087 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:33:25.606382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594496    5087 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:33:25.606382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594501    5087 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:33:25.606382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594506    5087 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:33:25.606382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594510    5087 flags.go:64] FLAG: --lock-file=""
Oct 30 21:33:25.606382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594515    5087 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:33:25.606382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594520    5087 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:33:25.606382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594526    5087 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:33:25.606382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594534    5087 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:33:25.606382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594539    5087 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:33:25.606382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594545    5087 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:33:25.606382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594550    5087 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:33:25.606382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594554    5087 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:33:25.606382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594559    5087 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:33:25.606382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594564    5087 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:33:25.606382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594569    5087 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:33:25.606382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594576    5087 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:33:25.606382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594580    5087 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:33:25.607413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594586    5087 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:33:25.607413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594591    5087 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:33:25.607413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594595    5087 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:33:25.607413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594600    5087 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:33:25.607413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594604    5087 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:33:25.607413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594609    5087 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:33:25.607413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594613    5087 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:33:25.607413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594618    5087 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:33:25.607413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594628    5087 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:33:25.607413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594632    5087 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:33:25.607413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594637    5087 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:33:25.607413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594642    5087 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:33:25.607413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594646    5087 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:33:25.607413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594650    5087 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:33:25.607413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594655    5087 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:33:25.607413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594659    5087 flags.go:64] FLAG: --port="10250"
Oct 30 21:33:25.607413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594664    5087 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:33:25.607413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594668    5087 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-05c15d85dc3eaf985"
Oct 30 21:33:25.607413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594673    5087 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:33:25.607413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594678    5087 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:33:25.607413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594682    5087 flags.go:64] FLAG: --register-node="true"
Oct 30 21:33:25.607413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594688    5087 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:33:25.607413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594697    5087 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:33:25.607413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594702    5087 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:33:25.607413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594707    5087 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:33:25.607413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594712    5087 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:33:25.608441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594718    5087 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:33:25.608441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594722    5087 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:33:25.608441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594727    5087 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:33:25.608441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594732    5087 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:33:25.608441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594737    5087 flags.go:64] FLAG: --runonce="false"
Oct 30 21:33:25.608441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594741    5087 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:33:25.608441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594746    5087 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:33:25.608441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594751    5087 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:33:25.608441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594755    5087 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:33:25.608441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594760    5087 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:33:25.608441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594765    5087 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:33:25.608441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594770    5087 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:33:25.608441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594775    5087 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:33:25.608441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594779    5087 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:33:25.608441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594784    5087 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:33:25.608441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594788    5087 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:33:25.608441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594793    5087 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:33:25.608441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594797    5087 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:33:25.608441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594802    5087 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:33:25.608441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594806    5087 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:33:25.608441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594814    5087 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:33:25.608441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594818    5087 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:33:25.608441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594823    5087 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:33:25.608441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594828    5087 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:33:25.608441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594832    5087 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:33:25.609511 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594837    5087 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:33:25.609511 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594841    5087 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:33:25.609511 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594847    5087 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:33:25.609511 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594852    5087 flags.go:64] FLAG: --v="2"
Oct 30 21:33:25.609511 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594859    5087 flags.go:64] FLAG: --version="false"
Oct 30 21:33:25.609511 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594865    5087 flags.go:64] FLAG: --vmodule=""
Oct 30 21:33:25.609511 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594871    5087 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:33:25.609511 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.594876    5087 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:33:25.609511 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595104    5087 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:33:25.609511 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595110    5087 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:33:25.609511 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595115    5087 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:33:25.609511 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595119    5087 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:33:25.609511 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595124    5087 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:33:25.609511 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595128    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:33:25.609511 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595132    5087 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:33:25.609511 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595136    5087 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:33:25.609511 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595140    5087 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:33:25.609511 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595145    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:33:25.609511 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595150    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:33:25.609511 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595154    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:33:25.609511 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595159    5087 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:33:25.609511 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595163    5087 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:33:25.610397 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595167    5087 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:33:25.610397 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595171    5087 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:33:25.610397 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595175    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:33:25.610397 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595179    5087 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:33:25.610397 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595183    5087 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:33:25.610397 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595187    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:33:25.610397 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595191    5087 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:33:25.610397 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595195    5087 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:33:25.610397 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595199    5087 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:33:25.610397 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595203    5087 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:33:25.610397 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595207    5087 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:33:25.610397 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595211    5087 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:33:25.610397 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595217    5087 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:33:25.610397 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595221    5087 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:33:25.610397 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595225    5087 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:33:25.610397 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595229    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:33:25.610397 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595233    5087 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:33:25.610397 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595239    5087 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:33:25.610397 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595245    5087 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:33:25.611137 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595250    5087 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:33:25.611137 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595256    5087 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:33:25.611137 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595260    5087 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:33:25.611137 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595264    5087 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:33:25.611137 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595268    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:33:25.611137 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595272    5087 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:33:25.611137 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595276    5087 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:33:25.611137 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595280    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:33:25.611137 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595284    5087 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:33:25.611137 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595289    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:33:25.611137 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595293    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:33:25.611137 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595297    5087 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:33:25.611137 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595301    5087 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:33:25.611137 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595307    5087 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:33:25.611137 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595313    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:33:25.611137 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595318    5087 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:33:25.611137 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595329    5087 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:33:25.611137 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595333    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:33:25.611137 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595337    5087 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:33:25.611981 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595342    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:33:25.611981 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595362    5087 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:33:25.611981 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595367    5087 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:33:25.611981 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595371    5087 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:33:25.611981 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595375    5087 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:33:25.611981 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595379    5087 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:33:25.611981 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595384    5087 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:33:25.611981 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595389    5087 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:33:25.611981 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595393    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:33:25.611981 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595397    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:33:25.611981 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595401    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:33:25.611981 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595405    5087 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:33:25.611981 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595409    5087 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:33:25.611981 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595414    5087 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:33:25.611981 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595419    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:33:25.611981 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595423    5087 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:33:25.611981 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595427    5087 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:33:25.611981 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595431    5087 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:33:25.611981 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595435    5087 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:33:25.611981 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595439    5087 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:33:25.612797 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595443    5087 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:33:25.612797 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595447    5087 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:33:25.612797 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595451    5087 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:33:25.612797 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595455    5087 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:33:25.612797 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595459    5087 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:33:25.612797 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595463    5087 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:33:25.612797 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595467    5087 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:33:25.612797 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595471    5087 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:33:25.612797 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595475    5087 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:33:25.612797 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595479    5087 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:33:25.612797 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595483    5087 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:33:25.612797 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595487    5087 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:33:25.612797 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595491    5087 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:33:25.612797 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595495    5087 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:33:25.612797 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595499    5087 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:33:25.612797 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595503    5087 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:33:25.612797 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595507    5087 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:33:25.612797 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595511    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:33:25.612797 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595516    5087 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:33:25.613591 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595520    5087 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:33:25.613591 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595524    5087 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:33:25.613591 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595529    5087 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:33:25.613591 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595533    5087 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:33:25.613591 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595537    5087 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:33:25.613591 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595541    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:33:25.613591 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595545    5087 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:33:25.613591 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595551    5087 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:33:25.613591 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595555    5087 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:33:25.613591 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595559    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:33:25.613591 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595563    5087 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:33:25.613591 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595567    5087 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:33:25.613591 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595571    5087 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:33:25.613591 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595575    5087 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:33:25.613591 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595580    5087 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:33:25.614188 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.595619    5087 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:33:25.614188 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595671    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:33:25.614188 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595678    5087 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:33:25.614188 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595682    5087 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:33:25.614188 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595686    5087 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:33:25.614188 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595690    5087 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:33:25.614188 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595694    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:33:25.614188 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595698    5087 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:33:25.614188 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595702    5087 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:33:25.614188 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595706    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:33:25.614188 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595710    5087 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:33:25.614188 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595714    5087 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:33:25.614188 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595718    5087 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:33:25.614188 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595722    5087 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:33:25.614188 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595726    5087 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:33:25.614188 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595730    5087 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:33:25.614837 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595735    5087 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:33:25.614837 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595739    5087 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:33:25.614837 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595743    5087 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:33:25.614837 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595748    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:33:25.614837 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595752    5087 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:33:25.614837 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595757    5087 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:33:25.614837 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595763    5087 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:33:25.614837 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595768    5087 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:33:25.614837 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595772    5087 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:33:25.614837 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595780    5087 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:33:25.614837 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595786    5087 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:33:25.614837 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595791    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:33:25.614837 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595794    5087 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:33:25.614837 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595799    5087 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:33:25.614837 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595803    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:33:25.614837 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595807    5087 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:33:25.614837 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595811    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:33:25.614837 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595815    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:33:25.614837 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595819    5087 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:33:25.615708 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595823    5087 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:33:25.615708 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595829    5087 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:33:25.615708 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595835    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:33:25.615708 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595839    5087 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:33:25.615708 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595844    5087 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:33:25.615708 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595848    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:33:25.615708 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595853    5087 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:33:25.615708 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595857    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:33:25.615708 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595861    5087 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:33:25.615708 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595865    5087 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:33:25.615708 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595869    5087 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:33:25.615708 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595873    5087 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:33:25.615708 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595877    5087 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:33:25.615708 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595882    5087 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:33:25.615708 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595886    5087 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:33:25.615708 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595890    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:33:25.615708 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595894    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:33:25.615708 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595898    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:33:25.615708 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595903    5087 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:33:25.616459 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595907    5087 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:33:25.616459 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595911    5087 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:33:25.616459 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595914    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:33:25.616459 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595921    5087 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:33:25.616459 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595925    5087 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:33:25.616459 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595929    5087 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:33:25.616459 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595934    5087 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:33:25.616459 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595938    5087 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:33:25.616459 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595942    5087 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:33:25.616459 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595946    5087 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:33:25.616459 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595949    5087 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:33:25.616459 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595953    5087 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:33:25.616459 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595957    5087 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:33:25.616459 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595961    5087 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:33:25.616459 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595965    5087 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:33:25.616459 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595969    5087 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:33:25.616459 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595973    5087 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:33:25.616459 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595977    5087 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:33:25.616459 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595981    5087 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:33:25.616459 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595985    5087 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:33:25.617297 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595989    5087 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:33:25.617297 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595993    5087 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:33:25.617297 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.595997    5087 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:33:25.617297 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.596001    5087 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:33:25.617297 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.596005    5087 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:33:25.617297 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.596009    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:33:25.617297 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.596016    5087 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:33:25.617297 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.596020    5087 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:33:25.617297 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.596024    5087 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:33:25.617297 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.596028    5087 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:33:25.617297 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.596032    5087 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:33:25.617297 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.596036    5087 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:33:25.617297 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.596040    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:33:25.617297 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.596044    5087 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:33:25.617297 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.596048    5087 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:33:25.617297 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.596053    5087 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:33:25.617297 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.596057    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:33:25.617297 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.596061    5087 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:33:25.617297 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.596065    5087 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:33:25.618063 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.596069    5087 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:33:25.618063 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.596073    5087 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:33:25.618063 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.596077    5087 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:33:25.618063 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.596082    5087 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:33:25.618063 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.596086    5087 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:33:25.618063 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.596090    5087 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:33:25.618063 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.596094    5087 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:33:25.618063 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.596099    5087 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:33:25.618063 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.596103    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:33:25.618063 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.596107    5087 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:33:25.618063 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.596111    5087 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:33:25.618063 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.596115    5087 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:33:25.618063 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.596119    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:33:25.618063 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.596123    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:33:25.618063 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.601619    5087 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:33:25.618063 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.601638    5087 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:33:25.618063 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.601661    5087 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:33:25.618063 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.601668    5087 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:33:25.618063 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601785    5087 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:33:25.618063 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601793    5087 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:33:25.619044 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601799    5087 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:33:25.619044 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601803    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:33:25.619044 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601808    5087 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:33:25.619044 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601814    5087 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:33:25.619044 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601819    5087 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:33:25.619044 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601824    5087 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:33:25.619044 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601828    5087 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:33:25.619044 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601833    5087 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:33:25.619044 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601838    5087 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:33:25.619044 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601843    5087 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:33:25.619044 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601854    5087 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:33:25.619044 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601858    5087 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:33:25.619044 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601862    5087 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:33:25.619044 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601866    5087 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:33:25.619044 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601870    5087 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:33:25.619044 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601874    5087 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:33:25.619044 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601878    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:33:25.619044 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601882    5087 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:33:25.619044 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601886    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:33:25.619044 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601890    5087 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:33:25.619876 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601895    5087 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:33:25.619876 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601899    5087 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:33:25.619876 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601904    5087 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:33:25.619876 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601908    5087 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:33:25.619876 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601912    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:33:25.619876 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601916    5087 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:33:25.619876 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601920    5087 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:33:25.619876 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601924    5087 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:33:25.619876 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601928    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:33:25.619876 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601932    5087 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:33:25.619876 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601937    5087 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:33:25.619876 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601941    5087 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:33:25.619876 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601945    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:33:25.619876 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601950    5087 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:33:25.619876 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601954    5087 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:33:25.619876 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601958    5087 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:33:25.619876 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601962    5087 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:33:25.619876 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601967    5087 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:33:25.619876 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601971    5087 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:33:25.620651 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601978    5087 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:33:25.620651 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601983    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:33:25.620651 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601989    5087 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:33:25.620651 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.601994    5087 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:33:25.620651 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602005    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:33:25.620651 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602010    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:33:25.620651 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602014    5087 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:33:25.620651 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602018    5087 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:33:25.620651 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602022    5087 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:33:25.620651 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602026    5087 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:33:25.620651 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602030    5087 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:33:25.620651 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602034    5087 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:33:25.620651 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602038    5087 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:33:25.620651 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602042    5087 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:33:25.620651 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602046    5087 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:33:25.620651 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602051    5087 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:33:25.620651 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602055    5087 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:33:25.620651 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602059    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:33:25.620651 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602064    5087 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:33:25.621421 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602068    5087 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:33:25.621421 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602072    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:33:25.621421 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602076    5087 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:33:25.621421 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602080    5087 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:33:25.621421 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602085    5087 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:33:25.621421 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602089    5087 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:33:25.621421 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602093    5087 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:33:25.621421 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602098    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:33:25.621421 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602102    5087 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:33:25.621421 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602107    5087 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:33:25.621421 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602111    5087 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:33:25.621421 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602115    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:33:25.621421 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602119    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:33:25.621421 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602123    5087 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:33:25.621421 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602127    5087 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:33:25.621421 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602131    5087 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:33:25.621421 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602136    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:33:25.621421 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602140    5087 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:33:25.621421 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602150    5087 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:33:25.622160 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602154    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:33:25.622160 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602158    5087 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:33:25.622160 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602162    5087 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:33:25.622160 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602166    5087 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:33:25.622160 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602170    5087 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:33:25.622160 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602174    5087 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:33:25.622160 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602178    5087 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:33:25.622160 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602183    5087 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:33:25.622160 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602187    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:33:25.622160 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602191    5087 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:33:25.622160 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602195    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:33:25.622160 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602199    5087 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:33:25.622160 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602203    5087 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:33:25.622160 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602207    5087 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:33:25.622160 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602211    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:33:25.622160 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602215    5087 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:33:25.622160 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602219    5087 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:33:25.622160 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602223    5087 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:33:25.622160 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602227    5087 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:33:25.622922 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602231    5087 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:33:25.622922 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602235    5087 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:33:25.622922 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602241    5087 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:33:25.622922 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602246    5087 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:33:25.622922 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602250    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:33:25.622922 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602254    5087 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:33:25.622922 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602258    5087 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:33:25.622922 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602262    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:33:25.622922 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.602301    5087 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:33:25.622922 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602590    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:33:25.622922 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602597    5087 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:33:25.622922 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602601    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:33:25.622922 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602605    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:33:25.622922 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602616    5087 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:33:25.622922 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602620    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:33:25.622922 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602625    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:33:25.624447 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602629    5087 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:33:25.624447 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602633    5087 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:33:25.624447 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602638    5087 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:33:25.624447 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602642    5087 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:33:25.624447 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602646    5087 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:33:25.624447 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602651    5087 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:33:25.624447 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602655    5087 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:33:25.624447 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602659    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:33:25.624447 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602663    5087 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:33:25.624447 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602667    5087 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:33:25.624447 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602671    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:33:25.624447 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602675    5087 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:33:25.624447 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602681    5087 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:33:25.624447 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602688    5087 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:33:25.624447 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602692    5087 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:33:25.624447 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602698    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:33:25.624447 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602702    5087 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:33:25.624447 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602706    5087 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:33:25.624447 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602710    5087 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:33:25.625199 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602714    5087 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:33:25.625199 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602718    5087 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:33:25.625199 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602723    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:33:25.625199 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602727    5087 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:33:25.625199 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602731    5087 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:33:25.625199 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602736    5087 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:33:25.625199 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602740    5087 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:33:25.625199 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602744    5087 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:33:25.625199 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602748    5087 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:33:25.625199 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602753    5087 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:33:25.625199 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602757    5087 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:33:25.625199 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602767    5087 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:33:25.625199 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602771    5087 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:33:25.625199 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602775    5087 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:33:25.625199 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602779    5087 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:33:25.625199 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602783    5087 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:33:25.625199 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602788    5087 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:33:25.625199 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602792    5087 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:33:25.625199 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602796    5087 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:33:25.625199 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602800    5087 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:33:25.626155 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602804    5087 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:33:25.626155 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602808    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:33:25.626155 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602812    5087 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:33:25.626155 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602816    5087 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:33:25.626155 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602820    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:33:25.626155 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602824    5087 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:33:25.626155 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602828    5087 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:33:25.626155 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602832    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:33:25.626155 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602836    5087 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:33:25.626155 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602840    5087 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:33:25.626155 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602846    5087 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:33:25.626155 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602851    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:33:25.626155 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602855    5087 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:33:25.626155 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602859    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:33:25.626155 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602867    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:33:25.626155 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602871    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:33:25.626155 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602875    5087 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:33:25.626155 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602879    5087 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:33:25.626155 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602884    5087 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:33:25.626975 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602888    5087 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:33:25.626975 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602892    5087 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:33:25.626975 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602896    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:33:25.626975 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602900    5087 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:33:25.626975 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602904    5087 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:33:25.626975 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602914    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:33:25.626975 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602919    5087 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:33:25.626975 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602923    5087 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:33:25.626975 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602929    5087 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:33:25.626975 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602933    5087 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:33:25.626975 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602937    5087 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:33:25.626975 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602941    5087 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:33:25.626975 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602945    5087 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:33:25.626975 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602949    5087 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:33:25.626975 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602953    5087 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:33:25.626975 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602957    5087 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:33:25.626975 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602960    5087 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:33:25.626975 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602964    5087 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:33:25.626975 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602968    5087 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:33:25.626975 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602972    5087 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:33:25.627804 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602976    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:33:25.627804 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602979    5087 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:33:25.627804 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602983    5087 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:33:25.627804 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602987    5087 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:33:25.627804 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602992    5087 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:33:25.627804 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.602996    5087 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:33:25.627804 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603000    5087 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:33:25.627804 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603006    5087 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:33:25.627804 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603010    5087 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:33:25.627804 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603014    5087 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:33:25.627804 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603018    5087 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:33:25.627804 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603022    5087 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:33:25.627804 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603026    5087 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:33:25.627804 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603031    5087 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:33:25.627804 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603035    5087 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:33:25.627804 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603039    5087 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:33:25.627804 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603043    5087 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:33:25.627804 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603053    5087 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:33:25.627804 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603064    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:33:25.627804 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603069    5087 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:33:25.628657 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603074    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:33:25.628657 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.603109    5087 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:33:25.628657 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603179    5087 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:33:25.628657 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603184    5087 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:33:25.628657 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603189    5087 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:33:25.628657 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603193    5087 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:33:25.628657 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603197    5087 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:33:25.628657 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603201    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:33:25.628657 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603205    5087 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:33:25.628657 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603209    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:33:25.628657 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603213    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:33:25.628657 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603218    5087 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:33:25.628657 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603222    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:33:25.628657 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603226    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:33:25.628657 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603230    5087 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:33:25.630474 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603234    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:33:25.630474 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603238    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:33:25.630474 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603242    5087 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:33:25.630474 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603246    5087 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:33:25.630474 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603251    5087 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:33:25.630474 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603256    5087 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:33:25.630474 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603260    5087 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:33:25.630474 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603264    5087 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:33:25.630474 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603268    5087 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:33:25.630474 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603272    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:33:25.630474 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603276    5087 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:33:25.630474 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603280    5087 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:33:25.630474 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603284    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:33:25.630474 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603288    5087 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:33:25.630474 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603294    5087 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:33:25.630474 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603300    5087 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:33:25.630474 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603311    5087 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:33:25.630474 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603317    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:33:25.630474 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603330    5087 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:33:25.631238 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603334    5087 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:33:25.631238 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603339    5087 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:33:25.631238 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603358    5087 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:33:25.631238 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603363    5087 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:33:25.631238 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603367    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:33:25.631238 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603371    5087 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:33:25.631238 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603375    5087 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:33:25.631238 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603380    5087 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:33:25.631238 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603384    5087 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:33:25.631238 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603388    5087 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:33:25.631238 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603392    5087 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:33:25.631238 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603397    5087 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:33:25.631238 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603401    5087 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:33:25.631238 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603405    5087 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:33:25.631238 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603410    5087 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:33:25.631238 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603414    5087 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:33:25.631238 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603418    5087 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:33:25.631238 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603422    5087 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:33:25.631238 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603428    5087 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:33:25.632071 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603432    5087 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:33:25.632071 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603436    5087 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:33:25.632071 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603440    5087 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:33:25.632071 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603445    5087 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:33:25.632071 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603449    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:33:25.632071 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603452    5087 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:33:25.632071 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603456    5087 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:33:25.632071 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603461    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:33:25.632071 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603464    5087 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:33:25.632071 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603468    5087 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:33:25.632071 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603472    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:33:25.632071 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603485    5087 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:33:25.632071 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603489    5087 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:33:25.632071 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603495    5087 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:33:25.632071 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603500    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:33:25.632071 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603505    5087 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:33:25.632071 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603509    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:33:25.632071 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603513    5087 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:33:25.632071 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603517    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:33:25.632849 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603521    5087 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:33:25.632849 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603525    5087 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:33:25.632849 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603529    5087 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:33:25.632849 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603534    5087 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:33:25.632849 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603538    5087 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:33:25.632849 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603542    5087 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:33:25.632849 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603546    5087 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:33:25.632849 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603550    5087 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:33:25.632849 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603553    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:33:25.632849 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603558    5087 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:33:25.632849 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603561    5087 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:33:25.632849 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603566    5087 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:33:25.632849 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603571    5087 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:33:25.632849 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603575    5087 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:33:25.632849 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603579    5087 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:33:25.632849 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603583    5087 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:33:25.632849 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603587    5087 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:33:25.632849 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603591    5087 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:33:25.632849 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603595    5087 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:33:25.632849 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603599    5087 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:33:25.633738 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603603    5087 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:33:25.633738 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603607    5087 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:33:25.633738 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603611    5087 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:33:25.633738 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603615    5087 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:33:25.633738 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603619    5087 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:33:25.633738 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603629    5087 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:33:25.633738 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603635    5087 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:33:25.633738 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603639    5087 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:33:25.633738 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603643    5087 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:33:25.633738 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603647    5087 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:33:25.633738 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603651    5087 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:33:25.633738 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603655    5087 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:33:25.633738 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603659    5087 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:33:25.633738 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603663    5087 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:33:25.633738 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603667    5087 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:33:25.633738 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:25.603671    5087 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:33:25.633738 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.603791    5087 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:33:25.633738 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.605298    5087 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 21:33:25.633738 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.605384    5087 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 21:33:25.634533 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.605692    5087 server.go:1014] "Starting client certificate rotation"
Oct 30 21:33:25.634533 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.606384    5087 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:33:25.634533 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.608095    5087 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:33:25.634533 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.608132    5087 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:27:43 +0000 UTC" deadline="2028-05-18 14:02:07.775000752 +0000 UTC"
Oct 30 21:33:25.634533 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.608183    5087 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13576h28m42.166820829s"
Oct 30 21:33:25.634533 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.608262    5087 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:33:25.634533 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.612604    5087 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:33:25.634533 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.619243    5087 log.go:25] "Validated CRI v1 image API"
Oct 30 21:33:25.634533 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.626058    5087 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:33:25.634533 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.629553    5087 fs.go:135] Filesystem UUIDs: map[248b64a8-55af-4ae5-8745-847a977ba628:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 fa2af845-2974-4011-b9d5-793eb5e1d844:/dev/nvme0n1p4]
Oct 30 21:33:25.634960 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.629578    5087 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0950186ea5083c75b1b9fae181e158056f873b387340151e7be3e063325afb53/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0950186ea5083c75b1b9fae181e158056f873b387340151e7be3e063325afb53/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ed851957e516501ee89a73862c5ac5353d106076614edd774fc99d801d9a022/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ed851957e516501ee89a73862c5ac5353d106076614edd774fc99d801d9a022/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b59cf8e851f9e4d60b0d55c5c8974801b568297de2f4f5b764aa9399df2eb83/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b59cf8e851f9e4d60b0d55c5c8974801b568297de2f4f5b764aa9399df2eb83/userdata/shm major:0 minor:215 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1deabd615573216006eaaa87f9f1473fe7c4c993b545315013612b8a44559d60/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1deabd615573216006eaaa87f9f1473fe7c4c993b545315013612b8a44559d60/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1e778827f2aa19da409b22ecbb0cc7a80f0e711817a34162237798e464a6c01e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1e778827f2aa19da409b22ecbb0cc7a80f0e711817a34162237798e464a6c01e/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/27149485ed43e2d13c16e47e539a93d2b28a9cb674c3d1bb83aedb22ee74a72e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/27149485ed43e2d13c16e47e539a93d2b28a9cb674c3d1bb83aedb22ee74a72e/userdata/shm major:0 minor:252 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2759ef767a46a495622a063dbe42fd1c49810f1d394f62f7605596f4fe4529f8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2759ef767a46a495622a063dbe42fd1c49810f1d394f62f7605596f4fe4529f8/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e73812d15ea38983aa08ff34298051b5b83a7ac0e2b02aa7ded043c23fddc62/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e73812d15ea38983aa08ff34298051b5b83a7ac0e2b02aa7ded043c23fddc62/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/31c6d1edb618126ab90cd9d1ced9d89feb0db514777f385fce55ca2540662aa0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/31c6d1edb618126ab90cd9d1ced9d89feb0db514777f385fce55ca2540662aa0/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/426c2d37a383fc0ad646fefef08f863879895c806c0dd0514b0e746ec625f730/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/426c2d37a383fc0ad646fefef08f863879895c806c0dd0514b0e746ec625f730/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/44128cde9864ff4f8f5f58931dcd7fa31cccfbe695c5adec39318d17a5bf4965/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/44128cde9864ff4f8f5f58931dcd7fa31cccfbe695c5adec39318d17a5bf4965/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/48b8e80d8364eab5f3806e15ec0306cbac919143c5cd2840df05403b1d771b95/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/48b8e80d8364eab5f3806e15ec0306cbac919143c5cd2840df05403b1d771b95/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4d36ec40de3b72acea76d6288acfc84460e0929e3022b70116f7c478ecabc504/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4d36ec40de3b72acea76d6288acfc84460e0929e3022b70116f7c478ecabc504/userdata/shm major:0 minor:212 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/578d4b23aaf1d1759ba22d8d449973b2451e249370addb2320c94042f416b02e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/578d4b23aaf1d1759ba22d8d449973b2451e249370addb2320c94042f416b02e/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/57a02ffca03a800228acf10ce6d2ebc4d848938007cfcf0eeeace4ae5152c715/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/57a02ffca03a800228acf10ce6d2ebc4d848938007cfcf0eeeace4ae5152c715/userdata/shm major:0 minor:52 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/76a2578487fd0958df5ea2687eadc05751d24772531eb6c223fcb41eecec2f52/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/76a2578487fd0958df5ea2687eadc05751d24772531eb6c223fcb41eecec2f52/userdata/shm major:0 minor:338 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/84fe6a4a113567dd158164901cf840c5c2f1e97eff7e29deeaea43687d082b21/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/84fe6a4a113567dd158164901cf840c5c2f1e97eff7e29deeaea43687d082b21/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/875aebd2b801f95612c15ae050e4703436aafd5316b77c6c2ce7a315981a29bd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/875aebd2b801f95612c15ae050e4703436aafd5316b77c6c2ce7a315981a29bd/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/88bd934e2000ef20455885675bf53af8a22f08bf75971d6fae1312c882bb3c38/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/88bd934e2000ef20455885675bf53af8a22f08bf75971d6fae1312c882bb3c38/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/916d64cb3b64d505ae30dc11ed921fcc439c6387395ff358fa55664fba9405af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/916d64cb3b64d505ae30dc11ed921fcc439c6387395ff358fa55664fba9405af/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ab1225b01af59fa63aff8166519a21119d51187887ff21d6c6fe4aedb3fbd04d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ab1225b01af59fa63aff8166519a21119d51187887ff21d6c6fe4aedb3fbd04d/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/add53b6f3bed35e52ef02565ebc5fe9d113ee6f5773c9203f1ff69f581149e0b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/add53b6f3bed35e52ef02565ebc5fe9d113ee6f5773c9203f1ff69f581149e0b/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aee7f142cb9248fa5e4c673b180f01bad80b209f93bccffc794f3473ec44cb1c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aee7f142cb9248fa5e4c673b180f01bad80b209f93bccffc794f3473ec44cb1c/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c3b73763545b02c62768710b9a3080c79703fc94c6cdbc27870033e6d52ec19c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c3b73763545b02c62768710b9a3080c79703fc94c6cdbc27870033e6d52ec19c/userdata/shm major:0 minor:362 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c3be87b2dc812051283450885198270cb1f6be3b7a2a5ed7a65389032a4fe0f8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c3be87b2dc812051283450885198270cb1f6be3b7a2a5ed7a65389032a4fe0f8/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ddb24eb1d1c6999a890779b2b932c9862c7f27eed0cd17007ad7d11bec1993f1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ddb24eb1d1c6999a890779b2b932c9862c7f27eed0cd17007ad7d11bec1993f1/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/032b9335-657e-4a86-aa6d-4ef46163014a/volumes/kubernetes.io~projected/kube-api-access-xjbfn:{mountpoint:/var/lib/kubelet/pods/032b9335-657e-4a86-aa6d-4ef46163014a/volumes/kubernetes.io~projected/kube-api-access-xjbfn major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/061313cb-4d50-4ae5-afda-dc63a62a013b/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/061313cb-4d50-4ae5-afda-dc63a62a013b/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/061313cb-4d50-4ae5-afda-dc63a62a013b/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/061313cb-4d50-4ae5-afda-dc63a62a013b/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/061313cb-4d50-4ae5-afda-dc63a62a013b/volumes/kubernetes.io~projected/kube-api-access-qdkhn:{mountpoint:/var/lib/kubelet/pods/061313cb-4d50-4ae5-afda-dc63a62a013b/volumes/kubernetes.io~projected/kube-api-access-qdkhn major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1196dab1-80d7-48eb-8d05-1a26ab6d0759/volumes/kubernetes.io~projected/kube-api-access-qrr6v:{mountpoint:/var/lib/kubelet/pods/1196dab1-80d7-48eb-8d05-1a26ab6d0759/volumes/kubernetes.io~projected/kube-api-access-qrr6v major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1196dab1-80d7-48eb-8d05-1a26ab6d0759/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/1196dab1-80d7-48eb-8d05-1a26ab6d0759/volumes/kubernetes.io~secret/serving-cert major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/239a77e8-635c-4388-ad79-1cb3b15129d8/volumes/kubernetes.io~projected/kube-api-access-t68x5:{mountpoint:/var/lib/kubelet/pods/239a77e8-635c-4388-ad79-1cb3b15129d8/volumes/kubernetes.io~projected/kube-api-access-t68x5 major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/23a040b8-2420-4d59-b426-f52832dc7de1/volumes/kubernetes.io~projected/kube-api-access-7rbpk:{mountpoint:/var/lib/kubelet/pods/23a040b8-2420-4d59-b426-f52832dc7de1/volumes/kubernetes.io~projected/kube-api-access-7rbpk major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245/volumes/kubernetes.io~projected/kube-api-access-tgn2z:{mountpoint:/var/lib/kubelet/pods/25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245/volumes/kubernetes.io~projected/kube-api-access-tgn2z major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245/volumes/kubernetes.io~secret/serving-cert major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/298f2a29-a322-4977-bd9f-d6b765eff5d1/volumes/kubernetes.io~projected/kube-api-access-ckxzq:{mountpoint:/var/lib/kubelet/pods/298f2a29-a322-4977-bd9f-d6b765eff5d1/volumes/kubernetes.io~projected/kube-api-access-ckxzq major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5/volumes/kubernetes.io~projected/kube-api-access-rq5j2:{mountpoint:/var/lib/kubelet/pods/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5/volumes/kubernetes.io~projected/kube-api-access-rq5j2 major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a7ef1f3-4e9d-40db-93a3-4d38a3474e89/volumes/kubernetes.io~projected/kube-api-access-kfn4f:{mountpoint:/var/lib/kubelet/pods/2a7ef1f3-4e9d-40db-93a3-4d38a3474e89/volumes/kubernetes.io~projected/kube-api-access-kfn4f major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/364111d6-7289-47be-8cf0-c10c7b6fbcdc/volumes/kubernetes.io~projected/kube-api-access-lzjqd:{mountpoint:/var/lib/kubelet/pods/364111d6-7289-47be-8cf0-c10c7b6fbcdc/volumes/kubernetes.io~projected/kube-api-access-lzjqd major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/392cc7ef-7cd2-4090-a246-0347d5af7bbf/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/392cc7ef-7cd2-4090-a246-0347d5af7bbf/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/392cc7ef-7cd2-4090-a246-0347d5af7bbf/volumes/kubernetes.io~projected/kube-api-access-f8z8h:{mountpoint:/var/lib/kubelet/pods/392cc7ef-7cd2-4090-a246-0347d5af7bbf/volumes/kubernetes.io~projected/kube-api-access-f8z8h major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/392cc7ef-7cd2-4090-a246-0347d5af7bbf/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/392cc7ef-7cd2-4090-a246-0347d5af7bbf/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/392cc7ef-7cd2-4090-a246-0347d5af7bbf/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/392cc7ef-7cd2-4090-a246-0347d5af7bbf/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ca424b7-c125-43ce-b837-88d1c02a05a0/volumes/kubernetes.io~projected/kube-api-access-jqsx2:{mountpoint:/var/lib/kubelet/pods/3ca424b7-c125-43ce-b837-88d1c02a05a0/volumes/kubernetes.io~projected/kube-api-access-jqsx2 major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3d0811a7-4a83-4758-8ca0-2ac7dd5a9408/volumes/kubernetes.io~projected/kube-api-access-h6gq8:{mountpoint:/var/lib/kubelet/pods/3d0811a7-4a83-4758-8ca0-2ac7dd5a9408/volumes/kubernetes.io~projected/kube-api-access-h6gq8 major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42dc1fb7-b6e0-40d2-bf53-20079815b12d/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/42dc1fb7-b6e0-40d2-bf53-20079815b12d/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42dc1fb7-b6e0-40d2-bf53-20079815b12d/volumes/kubernetes.io~projected/kube-api-access-77rbg:{mountpoint:/var/lib/kubelet/pods/42dc1fb7-b6e0-40d2-bf53-20079815b12d/volumes/kubernetes.io~projected/kube-api-access-77rbg major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42dc1fb7-b6e0-40d2-bf53-20079815b12d/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/42dc1fb7-b6e0-40d2-bf53-20079815b12d/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42dc1fb7-b6e0-40d2-bf53-20079815b12d/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/42dc1fb7-b6e0-40d2-bf53-20079815b12d/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5278b436-9321-4856-b9b0-ce9eb2596775/volumes/kubernetes.io~projected/kube-api-access-48g5f:{mountpoint:/var/lib/kubelet/pods/5278b436-9321-4856-b9b0-ce9eb2596775/volumes/kubernetes.io~projected/kube-api-access-48g5f major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/579a0c8f-9497-4448-98ab-4686adec22cd/volumes/kubernetes.io~projected/kube-api-access-6chm6:{mountpoint:/var/lib/kubelet/pods/579a0c8f-9497-4448-98ab-4686adec22cd/volumes/kubernetes.io~projected/kube-api-access-6chm6 major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e5648ef-56ca-4d99-ad1a-0adb0fc50855/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/6e5648ef-56ca-4d99-ad1a-0adb0fc50855/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/752a05d6-f630-42ec-9795-0d28e9ed7d80/volumes/kubernetes.io~projected/kube-api-access-qwsj7:{mountpoint:/var/lib/kubelet/pods/752a05d6-f630-42ec-9795-0d28e9ed7d80/volumes/kubernetes.io~projected/kube-api-access-qwsj7 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7986120b-482f-4b0b-a605-32770569962d/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/7986120b-482f-4b0b-a605-32770569962d/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7986120b-482f-4b0b-a605-32770569962d/volumes/kubernetes.io~projected/kube-api-access-pgwnq:{mountpoint:/var/lib/kubelet/pods/7986120b-482f-4b0b-a605-32770569962d/volumes/kubernetes.io~projected/kube-api-access-pgwnq major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7986120b-482f-4b0b-a605-32770569962d/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/7986120b-482f-4b0b-a605-32770569962d/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83917e8e-50b6-4c52-b337-6f1be5a2c26a/volumes/kubernetes.io~projected/kube-api-access-9wd6v:{mountpoint:/var/lib/kubelet/pods/83917e8e-50b6-4c52-b337-6f1be5a2c26a/volumes/kubernetes.io~projected/kube-api-access-9wd6v major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83917e8e-50b6-4c52-b337-6f1be5a2c26a/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/83917e8e-50b6-4c52-b337-6f1be5a2c26a/volumes/kubernetes.io~secret/default-certificate major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83917e8e-50b6-4c52-b337-6f1be5a2c26a/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/83917e8e-50b6-4c52-b337-6f1be5a2c26a/volumes/kubernetes.io~secret/stats-auth major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8a74b249-57da-4405-af8e-ae777a44415e/volumes/kubernetes.io~projected/kube-api-access-28w48:{mountpoint:/var/lib/kubelet/pods/8a74b249-57da-4405-af8e-ae777a44415e/volumes/kubernetes.io~projected/kube-api-access-28w48 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98a3aaba-8c58-43ae-837d-a809085d692c/volumes/kubernetes.io~projected/kube-api-access-4fbj2:{mountpoint:/var/lib/kubelet/pods/98a3aaba-8c58-43ae-837d-a809085d692c/volumes/kubernetes.io~projected/kube-api-access-4fbj2 major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/98a3aaba-8c58-43ae-837d-a809085d692c/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/98a3aaba-8c58-43ae-837d-a809085d692c/volumes/kubernetes.io~secret/signing-key major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a1460f5-20f5-4bd1-843f-3838935f2ce2/volumes/kubernetes.io~projected/kube-api-access-lkvjz:{mountpoint:/var/lib/kubelet/pods/9a1460f5-20f5-4bd1-843f-3838935f2ce2/volumes/kubernetes.io~projected/kube-api-access-lkvjz major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327/volumes/kubernetes.io~projected/kube-api-access-hk9wl:{mountpoint:/var/lib/kubelet/pods/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327/volumes/kubernetes.io~projected/kube-api-access-hk9wl major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a20999e6-4f67-47b9-9cf1-da1b1fb027f2/volumes/kubernetes.io~projected/kube-api-access-lkjl7:{mountpoint:/var/lib/kubelet/pods/a20999e6-4f67-47b9-9cf1-da1b1fb027f2/volumes/kubernetes.io~projected/kube-api-access-lkjl7 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a724a1c0-c643-49ff-a555-5bddec4bfacb/volumes/kubernetes.io~projected/kube-api-access-whb6q:{mountpoint:/var/lib/kubelet/pods/a724a1c0-c643-49ff-a555-5bddec4bfacb/volumes/kubernetes.io~projected/kube-api-access-whb6q major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5/volumes/kubernetes.io~projected/kube-api-access-4gw68:{mountpoint:/var/lib/kubelet/pods/b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5/volumes/kubernetes.io~projected/kube-api-access-4gw68 major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5/volumes/kubernetes.io~secret/cert major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e1a448f1-e180-4fbf-9923-fac0b9dbec74/volumes/kubernetes.io~projected/kube-api-access-xcz4j:{mountpoint:/var/lib/kubelet/pods/e1a448f1-e180-4fbf-9923-fac0b9dbec74/volumes/kubernetes.io~projected/kube-api-access-xcz4j major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f54f9754-2024-41b9-8f84-fc3d02a22253/volumes/kubernetes.io~projected/kube-api-access-tm65m:{mountpoint:/var/lib/kubelet/pods/f54f9754-2024-41b9-8f84-fc3d02a22253/volumes/kubernetes.io~projected/kube-api-access-tm65m major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f54f9754-2024-41b9-8f84-fc3d02a22253/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f54f9754-2024-41b9-8f84-fc3d02a22253/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f84984eb-c144-4394-bb77-afc0c94b3a3a/volumes/kubernetes.io~projected/kube-api-access-57sgv:{mountpoint:/var/lib/kubelet/pods/f84984eb-c144-4394-bb77-afc0c94b3a3a/volumes/kubernetes.io~projected/kube-api-access-57sgv major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf/volumes/kubernetes.io~projected/kube-api-access-p2mp5:{mountpoint:/var/lib/kubelet/pods/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf/volumes/kubernetes.io~projected/kube-api-access-p2mp5 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef/volumes/kubernetes.io~projected/kube-api-access-fbqkr:{mountpoint:/var/lib/kubelet/pods/f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef/volumes/kubernetes.io~projected/kube-api-access-fbqkr major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/907af8fadacbbb92cc8d6c65277a14961153ff55c6f7f16c85cddc565791d53f/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/43974c62ce14c4896d4781ee362a27bc3df86ad2632a6e3bbc97006f50ac63a8/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/4b1c18ddae67f30c2eaa988a80aa22c4f6db36075be7afb7631c5d309586d07a/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/659e1e00137b4c5513c26cf84a020a6c30603bd4aa2c4617202113907679e9b3/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/52a6c680706996f55a94edf5b4c2b8a1dca0907ad4e0d447e6f90203e6dc2702/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-117:{mountpoint:/var/lib/containers/storage/overlay/9f8a2872b80592498f930248a4a626ff1c410101406b1383c1d4f639489bce11/merged major:0 minor:117 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/84b39b879fbebc7003220d5f92ffff4be695cc9a537e735d4b8bbbc6a71e851e/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/5f91cd1487cba7d9c0a1434c6401081f084d8e474c390e1b98424dc1ac3fbd15/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/b6f6ebfd87e79862b1418e3557d26db4fcde124561a7cab4e0e527c52b86a3ff/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/d12e122ab23b7f30d2ca94262e7685a47bff1378a1cc8e8bf6626186b0b0a14e/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/2decd2f721c787aa2eb48e2a883584d0faaad46c7930fffd31fe4c2905c7eb24/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/a963df0fe7911aa2b03d0402ab004dc5cd2d6381b2e4a975e72f0a9dc2653d50/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/9e1131ac5270885bc3567556e52da1c359e3efdd38ca3fdcf0e1c11c4c09878a/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/67d5f41bfb10d2c3588e3c33deff2f458a30afb77d794f7e550ae72dfc7448cb/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/9e57e550fdd363f7c9cce85eaf19896dbf3eb48ed4738268cf6deca5d8b9acb9/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/9b1befd5c8c01e8f7b68d3900e568d776b3934b694b9fff8f298a4650eb485f9/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/cf2463bba200afb5467717467d532c92841f0fb9e0c220395028b83cf129f759/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/4b1973cc8cf91821692b71f3385fda172bef55659aa41b5d64ac1942b63cb7ab/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/1a99205e5e1e47c88e2943cf0e7db6a8484d1ff6535d895d6e02200504bc4c19/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/deea527a9a76466816b3f199370b44c84b356fa916f445b91799e65ab79d3b23/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/cbdce1cf0b6f4f90d0f0eba3b9b860d8d1a28ea5c6309992b3d7c75892128ed9/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/9c582179f720b806802998d4723df077ef3c69f81a69dc37b8e4b7b6ec0baca5/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/cfcbf151cedf569e9380c69ef82b12695520927222ad0a8d1132303719ab6214/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/7d4482b48afc85b57ae54acbd22c206803afe9a057f4b13ecd8b8c5d8c8aa818/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/5b5c2d5ed9e7bd66c8c18879125786b652d6a3f56af23d63941185c8f6dcb146/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/0c4ad85b5caacc9bd769f757cc70f0d06e55292947f96feef3fe5d235b2363a4/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/d6e03fdbead9ce5e33384887f8467fa1751798cf9b78360f9293de9809281199/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/a9911344b4647babb73af56f99f188895fbd3747dd460c164cc53b5a12d0ba8f/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/d7de701ce7a75146396c3489e2907943e0a699e54b825b60ccd861c352963150/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/ba63cb52ab55eac50d2784a8d459ca11aeacd7f8414693bc5714362f81e16c86/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/4764829fae1ef2e5e7587b2028e9950ff59c9a230bce09f50a9829fbf2986943/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/00020db0dde4cce595208de20344e0afd0ec389b23fe9eb3cf48e1a970cfd8ad/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/5ecb5c28bf4a8d72a082ca13fd5fdcd4fbf5f455bf393037c9bb7e8469227edf/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/5c3ae4a573bb9bfdae4b871e4bc324cec35ca2efe1422b2d1b82b417e07b39eb/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/ce9b5d99129434035de0a5888583204fa10bc15f5d82dfc0fc4a5c0a665f4b05/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/530f3d65d36cbf00ade69911181866fc34d2b6ade5e2c471cf400869356aa825/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/33740f7b040c48d5e5c72c0f4646aeff52cda07f245ce1bdc324af289a91c511/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/dea4d1d403c15b6c3d02de19a781e233a8673eb00d30ae70bb887d4ab0949ae5/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/671901b7746c14b217138db1c8c3785daeb4b8c2b4bda4a56db69ecb64a20a39/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/4cbeedf46e776a5428b0178dae13d81b9da4d8329ecb5b9ec171b4c778b4b666/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/4d8eabb455bdefa3b13d480a458ef1a936313b8819402a9e4d9e0f923afbf096/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/3d938bda61f84c4451a92f73c0c6d01f7bfe7d1e7b4a5683a0771e1fd1905f6c/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/8255aa9b6646575161046d74e1087b08954001b7a1bfdafee2515c1ab851d246/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/bfd7520e947ee429f4e98a88893ad12ad53e8a169189a076f5a13855cffd72e0/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/af077c2c07748d29d519580a50672d30eca4c03861106abeab9ea7b6552aa7f4/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/339411969ba678b0418e2c6d8675bac90e37cbd7e04f26928936d43d21819aca/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/75c6d3a2a7afbd08fbccc815a696545a45102913ac4a2bb60f1df489aac5e6d5/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/74a26e984f158d51c9b21a04fd67bb460ee6f50f7972c7ba3be8755b3dd38085/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/8c55828f5038135afe54efe661cba6a2460e2ea3585c3583e7a0c9160980cbb5/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/c5da7b1940f75ced2cd53051a1b0293e8243c5a2d171a08d4ec3a26fcc96a5aa/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/5fe5ec7fccb6eaa59ffa90c947da1510bb9aa60c3fb4987ad47d82d8f6e9c010/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/18a691dddea1b0db64958ae5dd7090335fe70e095805ea68198d3696cdf4cc01/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/37636ef2fe1832f9ac33de2bed1f5b38ff599e5811d9aafc2e6013b0e9124a59/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/576519dccebc15b124ab5108b0c2f0858b8e855b046db05e134c671fff47c77b/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/a05d3e00f18c97838c2a7ab718496ffd9129f5b2754b6f2c2d1512fc51916253/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/44e5fcd86bab94e7fa2127483a2adf965ed4922a3a30736939b967c0142fdfd1/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/c6390fd83893d6877a4febe76a3d3ad7a13ddf8b6e348eb44e9d0e50c751570c/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/4ae0607581c6813b092496393f45f9a819b2b72e0d019bb8dae0b562852f3e30/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/5a6cbd172b2f9fddbaf89fbbaf98a9ca2a5a460f0fbe5510b2eb9170542b8253/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/8da526c0eaf379324b2f454483f00350c8f1050903b9d9d1345cffc92f5b1d9f/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/330209b04252a60914d690c74917490095686c25d516d0618a32786385eab983/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/a7e21d6c867ce1fa74411df4560ad08f7fc77d7f5f5ef8b74e8e564c33bba226/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 21:33:25.642481 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.642080    5087 manager.go:217] Machine: {Timestamp:2026-10-30 21:33:25.641401049 +0000 UTC m=+0.115905026 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100034 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a4c1129a9e55f9492dc1481817b00 SystemUUID:ec2a4c11-29a9-e55f-9492-dc1481817b00 BootID:3a636b23-aa57-4caf-a687-6f46766e2cdc Filesystems:[{Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a7ef1f3-4e9d-40db-93a3-4d38a3474e89/volumes/kubernetes.io~projected/kube-api-access-kfn4f DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/42dc1fb7-b6e0-40d2-bf53-20079815b12d/volumes/kubernetes.io~projected/kube-api-access-77rbg DeviceMajor:0 DeviceMinor:228 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/44128cde9864ff4f8f5f58931dcd7fa31cccfbe695c5adec39318d17a5bf4965/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf/volumes/kubernetes.io~projected/kube-api-access-p2mp5 DeviceMajor:0 DeviceMinor:224 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/add53b6f3bed35e52ef02565ebc5fe9d113ee6f5773c9203f1ff69f581149e0b/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5278b436-9321-4856-b9b0-ce9eb2596775/volumes/kubernetes.io~projected/kube-api-access-48g5f DeviceMajor:0 DeviceMinor:334 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/23a040b8-2420-4d59-b426-f52832dc7de1/volumes/kubernetes.io~projected/kube-api-access-7rbpk DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef/volumes/kubernetes.io~projected/kube-api-access-fbqkr DeviceMajor:0 DeviceMinor:211 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1196dab1-80d7-48eb-8d05-1a26ab6d0759/volumes/kubernetes.io~projected/kube-api-access-qrr6v DeviceMajor:0 DeviceMinor:226 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/98a3aaba-8c58-43ae-837d-a809085d692c/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:356 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0ed851957e516501ee89a73862c5ac5353d106076614edd774fc99d801d9a022/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/426c2d37a383fc0ad646fefef08f863879895c806c0dd0514b0e746ec625f730/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/061313cb-4d50-4ae5-afda-dc63a62a013b/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f54f9754-2024-41b9-8f84-fc3d02a22253/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f84984eb-c144-4394-bb77-afc0c94b3a3a/volumes/kubernetes.io~projected/kube-api-access-57sgv DeviceMajor:0 DeviceMinor:209 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e1a448f1-e180-4fbf-9923-fac0b9dbec74/volumes/kubernetes.io~projected/kube-api-access-xcz4j DeviceMajor:0 DeviceMinor:287 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/032b9335-657e-4a86-aa6d-4ef46163014a/volumes/kubernetes.io~projected/kube-api-access-xjbfn DeviceMajor:0 DeviceMinor:54 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c3b73763545b02c62768710b9a3080c79703fc94c6cdbc27870033e6d52ec19c/userdata/shm DeviceMajor:0 DeviceMinor:362 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7986120b-482f-4b0b-a605-32770569962d/volumes/kubernetes.io~projected/kube-api-access-pgwnq DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/84fe6a4a113567dd158164901cf840c5c2f1e97eff7e29deeaea43687d082b21/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/83917e8e-50b6-4c52-b337-6f1be5a2c26a/volumes/kubernetes.io~projected/kube-api-access-9wd6v DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/392cc7ef-7cd2-4090-a246-0347d5af7bbf/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:216 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/42dc1fb7-b6e0-40d2-bf53-20079815b12d/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:218 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/579a0c8f-9497-4448-98ab-4686adec22cd/volumes/kubernetes.io~projected/kube-api-access-6chm6 DeviceMajor:0 DeviceMinor:238 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/061313cb-4d50-4ae5-afda-dc63a62a013b/volumes/kubernetes.io~projected/kube-api-access-qdkhn DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5/volumes/kubernetes.io~projected/kube-api-access-rq5j2 DeviceMajor:0 DeviceMinor:241 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/83917e8e-50b6-4c52-b337-6f1be5a2c26a/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/1196dab1-80d7-48eb-8d05-1a26ab6d0759/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:221 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/42dc1fb7-b6e0-40d2-bf53-20079815b12d/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:222 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:234 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/48b8e80d8364eab5f3806e15ec0306cbac919143c5cd2840df05403b1d771b95/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4d36ec40de3b72acea76d6288acfc84460e0929e3022b70116f7c478ecabc504/userdata/shm DeviceMajor:0 DeviceMinor:212 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/88bd934e2000ef20455885675bf53af8a22f08bf75971d6fae1312c882bb3c38/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/239a77e8-635c-4388-ad79-1cb3b15129d8/volumes/kubernetes.io~projected/kube-api-access-t68x5 DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0950186ea5083c75b1b9fae181e158056f873b387340151e7be3e063325afb53/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/298f2a29-a322-4977-bd9f-d6b765eff5d1/volumes/kubernetes.io~projected/kube-api-access-ckxzq DeviceMajor:0 DeviceMinor:333 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-117 DeviceMajor:0 DeviceMinor:117 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5/volumes/kubernetes.io~projected/kube-api-access-4gw68 DeviceMajor:0 DeviceMinor:240 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/364111d6-7289-47be-8cf0-c10c7b6fbcdc/volumes/kubernetes.io~projected/kube-api-access-lzjqd DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a724a1c0-c643-49ff-a555-5bddec4bfacb/volumes/kubernetes.io~projected/kube-api-access-whb6q DeviceMajor:0 DeviceMinor:328 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ca424b7-c125-43ce-b837-88d1c02a05a0/volumes/kubernetes.io~projected/kube-api-access-jqsx2 DeviceMajor:0 DeviceMinor:223 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6e5648ef-56ca-4d99-ad1a-0adb0fc50855/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/916d64cb3b64d505ae30dc11ed921fcc439c6387395ff358fa55664fba9405af/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2759ef767a46a495622a063dbe42fd1c49810f1d394f62f7605596f4fe4529f8/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1e778827f2aa19da409b22ecbb0cc7a80f0e711817a34162237798e464a6c01e/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327/volumes/kubernetes.io~projected/kube-api-access-hk9wl DeviceMajor:0 DeviceMinor:210 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:214 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/392cc7ef-7cd2-4090-a246-0347d5af7bbf/volumes/kubernetes.io~projected/kube-api-access-f8z8h DeviceMajor:0 DeviceMinor:227 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/061313cb-4d50-4ae5-afda-dc63a62a013b/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a20999e6-4f67-47b9-9cf1-da1b1fb027f2/volumes/kubernetes.io~projected/kube-api-access-lkjl7 DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3d0811a7-4a83-4758-8ca0-2ac7dd5a9408/volumes/kubernetes.io~projected/kube-api-access-h6gq8 DeviceMajor:0 DeviceMinor:335 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/31c6d1edb618126ab90cd9d1ced9d89feb0db514777f385fce55ca2540662aa0/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b59cf8e851f9e4d60b0d55c5c8974801b568297de2f4f5b764aa9399df2eb83/userdata/shm DeviceMajor:0 DeviceMinor:215 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/392cc7ef-7cd2-4090-a246-0347d5af7bbf/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:220 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aee7f142cb9248fa5e4c673b180f01bad80b209f93bccffc794f3473ec44cb1c/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f54f9754-2024-41b9-8f84-fc3d02a22253/volumes/kubernetes.io~projected/kube-api-access-tm65m DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/7986120b-482f-4b0b-a605-32770569962d/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e73812d15ea38983aa08ff34298051b5b83a7ac0e2b02aa7ded043c23fddc62/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245/volumes/kubernetes.io~projected/kube-api-access-tgn2z DeviceMajor:0 DeviceMinor:229 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ab1225b01af59fa63aff8166519a21119d51187887ff21d6c6fe4aedb3fbd04d/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/392cc7ef-7cd2-4090-a246-0347d5af7bbf/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:231 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/57a02ffca03a800228acf10ce6d2ebc4d848938007cfcf0eeeace4ae5152c715/userdata/shm DeviceMajor:0 DeviceMinor:52 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/875aebd2b801f95612c15ae050e4703436aafd5316b77c6c2ce7a315981a29bd/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8a74b249-57da-4405-af8e-ae777a44415e/volumes/kubernetes.io~projected/kube-api-access-28w48 DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/42dc1fb7-b6e0-40d2-bf53-20079815b12d/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:219 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/578d4b23aaf1d1759ba22d8d449973b2451e249370addb2320c94042f416b02e/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1deabd615573216006eaaa87f9f1473fe7c4c993b545315013612b8a44559d60/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c3be87b2dc812051283450885198270cb1f6be3b7a2a5ed7a65389032a4fe0f8/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/27149485ed43e2d13c16e47e539a93d2b28a9cb674c3d1bb83aedb22ee74a72e/userdata/shm DeviceMajor:0 DeviceMinor:252 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ddb24eb1d1c6999a890779b2b932c9862c7f27eed0cd17007ad7d11bec1993f1/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9a1460f5-20f5-4bd1-843f-3838935f2ce2/volumes/kubernetes.io~projected/kube-api-access-lkvjz DeviceMajor:0 DeviceMinor:53 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/83917e8e-50b6-4c52-b337-6f1be5a2c26a/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/76a2578487fd0958df5ea2687eadc05751d24772531eb6c223fcb41eecec2f52/userdata/shm DeviceMajor:0 DeviceMinor:338 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7986120b-482f-4b0b-a605-32770569962d/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/98a3aaba-8c58-43ae-837d-a809085d692c/volumes/kubernetes.io~projected/kube-api-access-4fbj2 DeviceMajor:0 DeviceMinor:361 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/752a05d6-f630-42ec-9795-0d28e9ed7d80/volumes/kubernetes.io~projected/kube-api-access-qwsj7 DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0950186ea5083c7 MacAddress:12:2b:2f:69:b7:ad Speed:10000 Mtu:8901} {Name:1b59cf8e851f9e4 MacAddress:0e:dd:91:f2:fa:a5 Speed:10000 Mtu:8901} {Name:1e778827f2aa19d MacAddress:de:88:ea:6b:02:f4 Speed:10000 Mtu:8901} {Name:27149485ed43e2d MacAddress:3a:26:8c:8c:81:de Speed:10000 Mtu:8901} {Name:2e73812d15ea389 MacAddress:3e:84:8e:f7:18:72 Speed:10000 Mtu:8901} {Name:31c6d1edb618126 MacAddress:7a:b8:7c:c6:fc:3b Speed:10000 Mtu:8901} {Name:426c2d37a383fc0 MacAddress:ce:43:23:71:2b:8e Speed:10000 Mtu:8901} {Name:4d36ec40de3b72a MacAddress:ca:f8:88:7f:78:a4 Speed:10000 Mtu:8901} {Name:76a2578487fd095 MacAddress:7e:86:73:85:61:e1 Speed:10000 Mtu:8901} {Name:88bd934e2000ef2 MacAddress:d2:10:d3:c4:65:f5 Speed:10000 Mtu:8901} {Name:916d64cb3b64d50 MacAddress:ee:7b:57:b4:e5:94 Speed:10000 Mtu:8901} {Name:ab1225b01af59fa MacAddress:de:8a:8b:81:6e:18 Speed:10000 Mtu:8901} {Name:add53b6f3bed35e MacAddress:4a:e1:bf:27:0c:d2 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:44:af:0d:e0:7b Speed:0 Mtu:9001} {Name:br-int MacAddress:ee:3a:9a:dc:ad:56 Speed:0 Mtu:8901} {Name:c3b73763545b02c MacAddress:46:13:eb:19:76:1d Speed:10000 Mtu:8901} {Name:c3be87b2dc81205 MacAddress:2e:bd:b0:19:e1:d0 Speed:10000 Mtu:8901} {Name:ddb24eb1d1c6999 MacAddress:ee:4f:b6:20:c9:bd Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:44:af:0d:e0:7b Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:56:a5:27:6c:39:55 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:be:8a:f6:0a:4f:3b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:33:25.642481 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.642485    5087 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:33:25.642860 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.642529    5087 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:33:25.642912 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.642853    5087 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:33:25.643015 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.642878    5087 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-92.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:33:25.643081 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.643016    5087 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:33:25.643081 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.643024    5087 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:33:25.643081 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.643037    5087 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:33:25.643081 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.643056    5087 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:33:25.643081 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.643075    5087 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:33:25.643287 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.643240    5087 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:33:25.643393 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.643382    5087 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:33:25.643441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.643422    5087 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:33:25.643441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.643437    5087 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:33:25.643523 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.643448    5087 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:33:25.643523 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.643457    5087 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:33:25.643523 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.643466    5087 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:33:25.645634 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.645618    5087 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:33:25.645687 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.645637    5087 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:33:25.646235 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.646216    5087 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:33:25.649228 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.649213    5087 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:33:25.649228 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.649227    5087 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:33:25.649333 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.649320    5087 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 21:33:25.649531 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.649518    5087 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:33:25.649581 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.649549    5087 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:33:25.649643 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.649634    5087 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:33:25.649686 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.649664    5087 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:33:25.649686 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.649675    5087 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:33:25.649686 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.649685    5087 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:33:25.649805 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.649691    5087 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:33:25.649805 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.649697    5087 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:33:25.649805 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.649703    5087 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:33:25.649805 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.649708    5087 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:33:25.649805 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.649714    5087 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:33:25.649805 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.649720    5087 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:33:25.649805 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.649733    5087 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:33:25.649805 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.649743    5087 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:33:25.649805 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.649771    5087 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:33:25.649805 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.649778    5087 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:33:25.653461 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.653446    5087 server.go:1263] "Started kubelet"
Oct 30 21:33:25.653677 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.653653    5087 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:33:25.653808 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.653754    5087 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:33:25.653868 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.653819    5087 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:33:25.653965 ip-10-0-132-92 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:33:25.654306 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.654289    5087 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:33:25.660774 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.660525    5087 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:33:25.664045 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.664024    5087 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:33:25.664318 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.664302    5087 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:33:25.664433 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.664411    5087 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:27:44 +0000 UTC" deadline="2028-06-06 12:50:36.161962763 +0000 UTC"
Oct 30 21:33:25.664485 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.664435    5087 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14031h17m10.4975305s"
Oct 30 21:33:25.664681 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.664580    5087 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:33:25.664681 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.664593    5087 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:33:25.664681 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.664662    5087 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:33:25.667474 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:25.667438    5087 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-92.us-west-1.compute.internal\" not found"
Oct 30 21:33:25.667556 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667432    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245" volumeName="kubernetes.io/secret/25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245-serving-cert" seLinuxMountContext=""
Oct 30 21:33:25.667623 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667560    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98a3aaba-8c58-43ae-837d-a809085d692c" volumeName="kubernetes.io/secret/98a3aaba-8c58-43ae-837d-a809085d692c-signing-key" seLinuxMountContext=""
Oct 30 21:33:25.667623 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667574    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="239a77e8-635c-4388-ad79-1cb3b15129d8" volumeName="kubernetes.io/projected/239a77e8-635c-4388-ad79-1cb3b15129d8-kube-api-access-t68x5" seLinuxMountContext=""
Oct 30 21:33:25.667623 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667586    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49" volumeName="kubernetes.io/secret/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-original-pull-secret" seLinuxMountContext=""
Oct 30 21:33:25.667623 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667599    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="392cc7ef-7cd2-4090-a246-0347d5af7bbf" volumeName="kubernetes.io/empty-dir/392cc7ef-7cd2-4090-a246-0347d5af7bbf-ca-trust-extracted" seLinuxMountContext=""
Oct 30 21:33:25.667623 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667611    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ca424b7-c125-43ce-b837-88d1c02a05a0" volumeName="kubernetes.io/projected/3ca424b7-c125-43ce-b837-88d1c02a05a0-kube-api-access-jqsx2" seLinuxMountContext=""
Oct 30 21:33:25.667841 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667623    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7986120b-482f-4b0b-a605-32770569962d" volumeName="kubernetes.io/projected/7986120b-482f-4b0b-a605-32770569962d-kube-api-access-pgwnq" seLinuxMountContext=""
Oct 30 21:33:25.667841 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667634    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a20999e6-4f67-47b9-9cf1-da1b1fb027f2" volumeName="kubernetes.io/projected/a20999e6-4f67-47b9-9cf1-da1b1fb027f2-kube-api-access-lkjl7" seLinuxMountContext=""
Oct 30 21:33:25.667841 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667648    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf" volumeName="kubernetes.io/configmap/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-config" seLinuxMountContext=""
Oct 30 21:33:25.667841 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667659    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="061313cb-4d50-4ae5-afda-dc63a62a013b" volumeName="kubernetes.io/projected/061313cb-4d50-4ae5-afda-dc63a62a013b-kube-api-access-qdkhn" seLinuxMountContext=""
Oct 30 21:33:25.667841 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667671    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="392cc7ef-7cd2-4090-a246-0347d5af7bbf" volumeName="kubernetes.io/secret/392cc7ef-7cd2-4090-a246-0347d5af7bbf-installation-pull-secrets" seLinuxMountContext=""
Oct 30 21:33:25.667841 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667682    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5278b436-9321-4856-b9b0-ce9eb2596775" volumeName="kubernetes.io/empty-dir/5278b436-9321-4856-b9b0-ce9eb2596775-archives-path" seLinuxMountContext=""
Oct 30 21:33:25.667841 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667692    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7986120b-482f-4b0b-a605-32770569962d" volumeName="kubernetes.io/configmap/7986120b-482f-4b0b-a605-32770569962d-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 21:33:25.667841 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667703    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f54f9754-2024-41b9-8f84-fc3d02a22253" volumeName="kubernetes.io/projected/f54f9754-2024-41b9-8f84-fc3d02a22253-kube-api-access-tm65m" seLinuxMountContext=""
Oct 30 21:33:25.667841 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667716    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a216b8f-a750-4e1e-9ab1-ff6c89d416e5" volumeName="kubernetes.io/configmap/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-config-volume" seLinuxMountContext=""
Oct 30 21:33:25.667841 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667727    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3d0811a7-4a83-4758-8ca0-2ac7dd5a9408" volumeName="kubernetes.io/projected/3d0811a7-4a83-4758-8ca0-2ac7dd5a9408-kube-api-access-h6gq8" seLinuxMountContext=""
Oct 30 21:33:25.667841 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667740    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42dc1fb7-b6e0-40d2-bf53-20079815b12d" volumeName="kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-bound-sa-token" seLinuxMountContext=""
Oct 30 21:33:25.667841 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667751    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="752a05d6-f630-42ec-9795-0d28e9ed7d80" volumeName="kubernetes.io/configmap/752a05d6-f630-42ec-9795-0d28e9ed7d80-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 21:33:25.667841 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667761    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a1460f5-20f5-4bd1-843f-3838935f2ce2" volumeName="kubernetes.io/configmap/9a1460f5-20f5-4bd1-843f-3838935f2ce2-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:33:25.667841 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667772    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1196dab1-80d7-48eb-8d05-1a26ab6d0759" volumeName="kubernetes.io/configmap/1196dab1-80d7-48eb-8d05-1a26ab6d0759-trusted-ca" seLinuxMountContext=""
Oct 30 21:33:25.667841 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667783    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42dc1fb7-b6e0-40d2-bf53-20079815b12d" volumeName="kubernetes.io/configmap/42dc1fb7-b6e0-40d2-bf53-20079815b12d-trusted-ca" seLinuxMountContext=""
Oct 30 21:33:25.667841 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667794    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5278b436-9321-4856-b9b0-ce9eb2596775" volumeName="kubernetes.io/projected/5278b436-9321-4856-b9b0-ce9eb2596775-kube-api-access-48g5f" seLinuxMountContext=""
Oct 30 21:33:25.667841 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667806    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="752a05d6-f630-42ec-9795-0d28e9ed7d80" volumeName="kubernetes.io/configmap/752a05d6-f630-42ec-9795-0d28e9ed7d80-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:33:25.667841 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667817    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f54f9754-2024-41b9-8f84-fc3d02a22253" volumeName="kubernetes.io/configmap/f54f9754-2024-41b9-8f84-fc3d02a22253-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 21:33:25.667841 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667829    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f54f9754-2024-41b9-8f84-fc3d02a22253" volumeName="kubernetes.io/secret/f54f9754-2024-41b9-8f84-fc3d02a22253-serving-cert" seLinuxMountContext=""
Oct 30 21:33:25.667841 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667842    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="032b9335-657e-4a86-aa6d-4ef46163014a" volumeName="kubernetes.io/projected/032b9335-657e-4a86-aa6d-4ef46163014a-kube-api-access-xjbfn" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667854    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e5648ef-56ca-4d99-ad1a-0adb0fc50855" volumeName="kubernetes.io/secret/6e5648ef-56ca-4d99-ad1a-0adb0fc50855-agent-certs" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667866    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="752a05d6-f630-42ec-9795-0d28e9ed7d80" volumeName="kubernetes.io/projected/752a05d6-f630-42ec-9795-0d28e9ed7d80-kube-api-access-qwsj7" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667882    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83917e8e-50b6-4c52-b337-6f1be5a2c26a" volumeName="kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-stats-auth" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667893    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a1460f5-20f5-4bd1-843f-3838935f2ce2" volumeName="kubernetes.io/configmap/9a1460f5-20f5-4bd1-843f-3838935f2ce2-multus-daemon-config" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667904    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5" volumeName="kubernetes.io/secret/b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5-cert" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667915    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f54f9754-2024-41b9-8f84-fc3d02a22253" volumeName="kubernetes.io/configmap/f54f9754-2024-41b9-8f84-fc3d02a22253-service-ca-bundle" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667926    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf" volumeName="kubernetes.io/projected/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-kube-api-access-p2mp5" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667937    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23a040b8-2420-4d59-b426-f52832dc7de1" volumeName="kubernetes.io/projected/23a040b8-2420-4d59-b426-f52832dc7de1-kube-api-access-7rbpk" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667948    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="364111d6-7289-47be-8cf0-c10c7b6fbcdc" volumeName="kubernetes.io/projected/364111d6-7289-47be-8cf0-c10c7b6fbcdc-kube-api-access-lzjqd" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667958    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5278b436-9321-4856-b9b0-ce9eb2596775" volumeName="kubernetes.io/configmap/5278b436-9321-4856-b9b0-ce9eb2596775-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667969    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98a3aaba-8c58-43ae-837d-a809085d692c" volumeName="kubernetes.io/configmap/98a3aaba-8c58-43ae-837d-a809085d692c-signing-cabundle" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667981    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42dc1fb7-b6e0-40d2-bf53-20079815b12d" volumeName="kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-kube-api-access-77rbg" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.667992    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a724a1c0-c643-49ff-a555-5bddec4bfacb" volumeName="kubernetes.io/empty-dir/a724a1c0-c643-49ff-a555-5bddec4bfacb-data-volume" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668002    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef" volumeName="kubernetes.io/configmap/f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef-config" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668013    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a216b8f-a750-4e1e-9ab1-ff6c89d416e5" volumeName="kubernetes.io/projected/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-kube-api-access-rq5j2" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668024    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="392cc7ef-7cd2-4090-a246-0347d5af7bbf" volumeName="kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-kube-api-access-f8z8h" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668034    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="579a0c8f-9497-4448-98ab-4686adec22cd" volumeName="kubernetes.io/projected/579a0c8f-9497-4448-98ab-4686adec22cd-kube-api-access-6chm6" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668045    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83917e8e-50b6-4c52-b337-6f1be5a2c26a" volumeName="kubernetes.io/projected/83917e8e-50b6-4c52-b337-6f1be5a2c26a-kube-api-access-9wd6v" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668055    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5" volumeName="kubernetes.io/projected/b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5-kube-api-access-4gw68" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668066    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d5f25065-34c9-4724-8dff-9ace0890d565" volumeName="kubernetes.io/configmap/d5f25065-34c9-4724-8dff-9ace0890d565-nginx-conf" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668077    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e1a448f1-e180-4fbf-9923-fac0b9dbec74" volumeName="kubernetes.io/projected/e1a448f1-e180-4fbf-9923-fac0b9dbec74-kube-api-access-xcz4j" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668109    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245" volumeName="kubernetes.io/projected/25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245-kube-api-access-tgn2z" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668122    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="298f2a29-a322-4977-bd9f-d6b765eff5d1" volumeName="kubernetes.io/projected/298f2a29-a322-4977-bd9f-d6b765eff5d1-kube-api-access-ckxzq" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668135    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7986120b-482f-4b0b-a605-32770569962d" volumeName="kubernetes.io/configmap/7986120b-482f-4b0b-a605-32770569962d-ovnkube-config" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668149    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f16bd1f-a3bd-45ee-a02e-cb6b9e982327" volumeName="kubernetes.io/configmap/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-telemetry-config" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668160    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef" volumeName="kubernetes.io/secret/f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef-serving-cert" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668172    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="061313cb-4d50-4ae5-afda-dc63a62a013b" volumeName="kubernetes.io/empty-dir/061313cb-4d50-4ae5-afda-dc63a62a013b-tmp" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668184    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a216b8f-a750-4e1e-9ab1-ff6c89d416e5" volumeName="kubernetes.io/empty-dir/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-tmp-dir" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668195    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f16bd1f-a3bd-45ee-a02e-cb6b9e982327" volumeName="kubernetes.io/projected/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-kube-api-access-hk9wl" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668207    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="239a77e8-635c-4388-ad79-1cb3b15129d8" volumeName="kubernetes.io/empty-dir/239a77e8-635c-4388-ad79-1cb3b15129d8-tmp-dir" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668226    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a1460f5-20f5-4bd1-843f-3838935f2ce2" volumeName="kubernetes.io/projected/9a1460f5-20f5-4bd1-843f-3838935f2ce2-kube-api-access-lkvjz" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668238    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1196dab1-80d7-48eb-8d05-1a26ab6d0759" volumeName="kubernetes.io/projected/1196dab1-80d7-48eb-8d05-1a26ab6d0759-kube-api-access-qrr6v" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668250    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="752a05d6-f630-42ec-9795-0d28e9ed7d80" volumeName="kubernetes.io/configmap/752a05d6-f630-42ec-9795-0d28e9ed7d80-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668270    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7986120b-482f-4b0b-a605-32770569962d" volumeName="kubernetes.io/configmap/7986120b-482f-4b0b-a605-32770569962d-env-overrides" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668280    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8a74b249-57da-4405-af8e-ae777a44415e" volumeName="kubernetes.io/projected/8a74b249-57da-4405-af8e-ae777a44415e-kube-api-access-28w48" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668292    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a20999e6-4f67-47b9-9cf1-da1b1fb027f2" volumeName="kubernetes.io/configmap/a20999e6-4f67-47b9-9cf1-da1b1fb027f2-serviceca" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668302    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a724a1c0-c643-49ff-a555-5bddec4bfacb" volumeName="kubernetes.io/projected/a724a1c0-c643-49ff-a555-5bddec4bfacb-kube-api-access-whb6q" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668313    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f84984eb-c144-4394-bb77-afc0c94b3a3a" volumeName="kubernetes.io/projected/f84984eb-c144-4394-bb77-afc0c94b3a3a-kube-api-access-57sgv" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668325    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a7ef1f3-4e9d-40db-93a3-4d38a3474e89" volumeName="kubernetes.io/configmap/2a7ef1f3-4e9d-40db-93a3-4d38a3474e89-iptables-alerter-script" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668337    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="392cc7ef-7cd2-4090-a246-0347d5af7bbf" volumeName="kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-bound-sa-token" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668363    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42dc1fb7-b6e0-40d2-bf53-20079815b12d" volumeName="kubernetes.io/configmap/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-certificates" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668374    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42dc1fb7-b6e0-40d2-bf53-20079815b12d" volumeName="kubernetes.io/empty-dir/42dc1fb7-b6e0-40d2-bf53-20079815b12d-ca-trust-extracted" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668384    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5278b436-9321-4856-b9b0-ce9eb2596775" volumeName="kubernetes.io/configmap/5278b436-9321-4856-b9b0-ce9eb2596775-service-ca-bundle" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668395    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7986120b-482f-4b0b-a605-32770569962d" volumeName="kubernetes.io/secret/7986120b-482f-4b0b-a605-32770569962d-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668405    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f54f9754-2024-41b9-8f84-fc3d02a22253" volumeName="kubernetes.io/empty-dir/f54f9754-2024-41b9-8f84-fc3d02a22253-tmp" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668415    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef" volumeName="kubernetes.io/projected/f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef-kube-api-access-fbqkr" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668426    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a7ef1f3-4e9d-40db-93a3-4d38a3474e89" volumeName="kubernetes.io/projected/2a7ef1f3-4e9d-40db-93a3-4d38a3474e89-kube-api-access-kfn4f" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668437    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="392cc7ef-7cd2-4090-a246-0347d5af7bbf" volumeName="kubernetes.io/configmap/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-certificates" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668448    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="392cc7ef-7cd2-4090-a246-0347d5af7bbf" volumeName="kubernetes.io/secret/392cc7ef-7cd2-4090-a246-0347d5af7bbf-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668462    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e5648ef-56ca-4d99-ad1a-0adb0fc50855" volumeName="kubernetes.io/configmap/6e5648ef-56ca-4d99-ad1a-0adb0fc50855-konnectivity-ca" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668474    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f54f9754-2024-41b9-8f84-fc3d02a22253" volumeName="kubernetes.io/empty-dir/f54f9754-2024-41b9-8f84-fc3d02a22253-snapshots" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668485    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="061313cb-4d50-4ae5-afda-dc63a62a013b" volumeName="kubernetes.io/empty-dir/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-tuned" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668496    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1196dab1-80d7-48eb-8d05-1a26ab6d0759" volumeName="kubernetes.io/secret/1196dab1-80d7-48eb-8d05-1a26ab6d0759-serving-cert" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668508    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245" volumeName="kubernetes.io/configmap/25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245-config" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668520    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="392cc7ef-7cd2-4090-a246-0347d5af7bbf" volumeName="kubernetes.io/configmap/392cc7ef-7cd2-4090-a246-0347d5af7bbf-trusted-ca" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668531    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42dc1fb7-b6e0-40d2-bf53-20079815b12d" volumeName="kubernetes.io/secret/42dc1fb7-b6e0-40d2-bf53-20079815b12d-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668543    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42dc1fb7-b6e0-40d2-bf53-20079815b12d" volumeName="kubernetes.io/secret/42dc1fb7-b6e0-40d2-bf53-20079815b12d-installation-pull-secrets" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668561    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83917e8e-50b6-4c52-b337-6f1be5a2c26a" volumeName="kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-default-certificate" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668572    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="98a3aaba-8c58-43ae-837d-a809085d692c" volumeName="kubernetes.io/projected/98a3aaba-8c58-43ae-837d-a809085d692c-kube-api-access-4fbj2" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668584    5087 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1196dab1-80d7-48eb-8d05-1a26ab6d0759" volumeName="kubernetes.io/configmap/1196dab1-80d7-48eb-8d05-1a26ab6d0759-config" seLinuxMountContext=""
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668595    5087 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:33:25.668678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.668602    5087 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:33:25.673645 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.673617    5087 factory.go:55] Registering systemd factory
Oct 30 21:33:25.673715 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.673648    5087 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:33:25.675541 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.675528    5087 factory.go:153] Registering CRI-O factory
Oct 30 21:33:25.675635 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.675627    5087 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:33:25.675766 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.675757    5087 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:33:25.675850 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.675842    5087 factory.go:103] Registering Raw factory
Oct 30 21:33:25.675916 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.675909    5087 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:33:25.676277 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.676268    5087 manager.go:319] Starting recovery of all containers
Oct 30 21:33:25.696398 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.696313    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 21:33:25.697637 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.697610    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 21:33:25.697637 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.697617    5087 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:33:25.700957 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.700934    5087 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:33:25.700957 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.700958    5087 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:33:25.701076 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.700978    5087 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:33:25.701076 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:25.701020    5087 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 21:33:25.715077 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.715057    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 21:33:25.720028 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.720002    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 21:33:25.734696 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.734679    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 21:33:25.735043 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.735011    5087 generic.go:356] "Generic (PLEG): container finished" podID="7986120b-482f-4b0b-a605-32770569962d" containerID="7fa32e4295690d9ce92762231369af00c9e3bd979b045ddeb844417b0dc5c82c" exitCode=1
Oct 30 21:33:25.757630 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.757595    5087 generic.go:356] "Generic (PLEG): container finished" podID="04840b9eb1acb1deca2b80fddf33a1d0" containerID="ffe8166c062f1d8279eba9d50be23880cab39bb109adfbd71b590c4f386710e0" exitCode=0
Oct 30 21:33:25.763607 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.763587    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 21:33:25.763705 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.763616    5087 generic.go:356] "Generic (PLEG): container finished" podID="1196dab1-80d7-48eb-8d05-1a26ab6d0759" containerID="d036f501bf39148b5ef1dfc318a42d74c69b78f8ea89630deb12c0797ede2521" exitCode=255
Oct 30 21:33:25.767530 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:25.767512    5087 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-92.us-west-1.compute.internal\" not found"
Oct 30 21:33:25.770392 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.770368    5087 generic.go:356] "Generic (PLEG): container finished" podID="752a05d6-f630-42ec-9795-0d28e9ed7d80" containerID="045d16bc1eb4a855c2057cdeed55436899647a11ebc902b6bbabde8f04dc4eb7" exitCode=0
Oct 30 21:33:25.770482 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.770392    5087 generic.go:356] "Generic (PLEG): container finished" podID="752a05d6-f630-42ec-9795-0d28e9ed7d80" containerID="865fab4defdcaa4ee2132c939d8efc22b43d74ab7da4f6812aa24c9beac08cf1" exitCode=0
Oct 30 21:33:25.770482 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.770405    5087 generic.go:356] "Generic (PLEG): container finished" podID="752a05d6-f630-42ec-9795-0d28e9ed7d80" containerID="31916e0f83c7a5cc25d29dff3528113f22c5ac455bc8260b9c0ed0cc2a572207" exitCode=0
Oct 30 21:33:25.770482 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.770415    5087 generic.go:356] "Generic (PLEG): container finished" podID="752a05d6-f630-42ec-9795-0d28e9ed7d80" containerID="5b0fc9b4dfae26dd49b535ae3d39287c0636d60510c6eac1c7c3e1b176f5320a" exitCode=0
Oct 30 21:33:25.770482 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.770423    5087 generic.go:356] "Generic (PLEG): container finished" podID="752a05d6-f630-42ec-9795-0d28e9ed7d80" containerID="33c75b63ac9e8b43036f5be84f365f19874f7b06165f267279e372fdcc2e4cd6" exitCode=0
Oct 30 21:33:25.770482 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.770431    5087 generic.go:356] "Generic (PLEG): container finished" podID="752a05d6-f630-42ec-9795-0d28e9ed7d80" containerID="ad5abbc3081b61b6528a8247fa6863218e1a950a079381330b84c53d98cf6240" exitCode=0
Oct 30 21:33:25.779895 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.779881    5087 manager.go:324] Recovery completed
Oct 30 21:33:25.801091 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:25.801069    5087 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 30 21:33:25.801405 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.801391    5087 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:33:25.802543 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.802529    5087 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:33:25.802591 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.802556    5087 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:33:25.802591 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.802566    5087 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:33:25.804574 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.804558    5087 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:33:25.804574 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.804572    5087 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:33:25.804678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.804573    5087 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:33:25.804678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.804589    5087 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:33:25.804759 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.804699    5087 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 21:33:25.804759 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.804710    5087 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 21:33:25.804759 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.804726    5087 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 21:33:25.804759 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.804731    5087 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 21:33:25.804759 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.804737    5087 policy_none.go:50] "Start"
Oct 30 21:33:25.804759 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.804745    5087 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:33:25.804759 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.804756    5087 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:33:25.804961 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.804888    5087 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 21:33:25.804961 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.804895    5087 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 21:33:25.804961 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.804901    5087 policy_none.go:44] "Start"
Oct 30 21:33:25.805499 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.805484    5087 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:33:25.805554 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.805511    5087 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:33:25.805554 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.805523    5087 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:33:25.809653 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.809627    5087 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:33:25.812301 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.812290    5087 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:33:25.812388 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:25.812311    5087 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:33:25.812388 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.812320    5087 server.go:87] "Starting device plugin registration server"
Oct 30 21:33:25.812592 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.812582    5087 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:33:25.812624 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.812593    5087 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:33:25.812721 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.812702    5087 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:33:25.812839 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.812811    5087 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:33:25.812891 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.812848    5087 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:33:25.812891 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.812855    5087 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:33:25.813461 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.812913    5087 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:33:25.812837144Z","UUID":"5b4681d5-4d81-4618-a740-cc25312acc74","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:33:25.814465 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:25.814438    5087 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:33:25.814558 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:25.814468    5087 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-92.us-west-1.compute.internal\" not found"
Oct 30 21:33:25.814558 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.814475    5087 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:33:25.814558 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.814499    5087 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:33:25.850160 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.850139    5087 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:33:25.850314 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.850295    5087 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:33:25.850386 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.850308    5087 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 21:33:25.899447 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.899417    5087 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:33:25.902118 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.902092    5087 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:33:25.912741 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.912721    5087 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:33:25.913978 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.913959    5087 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:33:25.914147 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.914010    5087 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:33:25.914147 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.914025    5087 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-92.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:33:25.914147 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:25.914083    5087 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:33:26.001642 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.001592    5087 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal"]
Oct 30 21:33:26.002257 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.002226    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal" event={"ID":"57c3091500d45dc621d6409761c7c97d","Type":"ContainerStarted","Data":"3b2b55c7064b47a34cf4c24d5fbd4172bc700f8cd92a5c1ea9ed0b65222106a1"}
Oct 30 21:33:26.002326 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.002265    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal" event={"ID":"57c3091500d45dc621d6409761c7c97d","Type":"ContainerStarted","Data":"2759ef767a46a495622a063dbe42fd1c49810f1d394f62f7605596f4fe4529f8"}
Oct 30 21:33:26.002326 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.002278    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal" event={"ID":"04840b9eb1acb1deca2b80fddf33a1d0","Type":"ContainerStarted","Data":"108841fd68a8ca97ffb4121c526d3bd00f35b3a91ced5fbdb275db35fba7126f"}
Oct 30 21:33:26.002326 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.002290    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal" event={"ID":"04840b9eb1acb1deca2b80fddf33a1d0","Type":"ContainerDied","Data":"ffe8166c062f1d8279eba9d50be23880cab39bb109adfbd71b590c4f386710e0"}
Oct 30 21:33:26.002326 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.002303    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal" event={"ID":"04840b9eb1acb1deca2b80fddf33a1d0","Type":"ContainerStarted","Data":"57a02ffca03a800228acf10ce6d2ebc4d848938007cfcf0eeeace4ae5152c715"}
Oct 30 21:33:26.002487 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.002342    5087 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:33:26.002487 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.002415    5087 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:33:26.022390 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.022367    5087 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:33:26.022502 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.022420    5087 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:33:26.062676 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.062647    5087 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:33:26.062807 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:26.062702    5087 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:33:26.063587 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.063565    5087 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:33:26.063697 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:26.063606    5087 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:33:26.152176 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.152145    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04840b9eb1acb1deca2b80fddf33a1d0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal\" (UID: \"04840b9eb1acb1deca2b80fddf33a1d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:33:26.152176 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.152175    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57c3091500d45dc621d6409761c7c97d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal\" (UID: \"57c3091500d45dc621d6409761c7c97d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:33:26.152378 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.152193    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/04840b9eb1acb1deca2b80fddf33a1d0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal\" (UID: \"04840b9eb1acb1deca2b80fddf33a1d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:33:26.253402 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.253317    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/04840b9eb1acb1deca2b80fddf33a1d0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal\" (UID: \"04840b9eb1acb1deca2b80fddf33a1d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:33:26.253402 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.253368    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04840b9eb1acb1deca2b80fddf33a1d0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal\" (UID: \"04840b9eb1acb1deca2b80fddf33a1d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:33:26.253568 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.253421    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04840b9eb1acb1deca2b80fddf33a1d0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal\" (UID: \"04840b9eb1acb1deca2b80fddf33a1d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:33:26.253568 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.253428    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/04840b9eb1acb1deca2b80fddf33a1d0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal\" (UID: \"04840b9eb1acb1deca2b80fddf33a1d0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:33:26.253568 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.253456    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57c3091500d45dc621d6409761c7c97d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal\" (UID: \"57c3091500d45dc621d6409761c7c97d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:33:26.253568 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.253484    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/57c3091500d45dc621d6409761c7c97d-config\") pod \"kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal\" (UID: \"57c3091500d45dc621d6409761c7c97d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:33:26.644486 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.644420    5087 apiserver.go:52] "Watching apiserver"
Oct 30 21:33:26.774065 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.774035    5087 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:33:26.774224 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.774185    5087 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:33:26.799038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.799008    5087 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:33:26.799824 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.799795    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-cx7d4","kube-system/kas-connection-checker-685895c7c9-9trcs","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j","openshift-network-diagnostics/network-check-source-6f78d59f99-64b9m","openshift-console-operator/console-operator-7cd56b8dbc-z8clf","openshift-service-ca-operator/service-ca-operator-544d749c8c-bn8fx","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc","openshift-ingress/router-default-54cbfc65d5-sg48h","openshift-cluster-node-tuning-operator/tuned-86426","openshift-insights/insights-runtime-extractor-g27zs","kube-system/kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal","openshift-multus/multus-25lhq","openshift-insights/insights-operator-7c4986d449-z4jbc","openshift-dns/dns-default-5kfvz","openshift-kube-storage-version-migrator/migrator-66df5d496b-fjrtf","openshift-kube-storage-version-migrator/migrator-66df5d496b-mf2wv","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd","openshift-ovn-kubernetes/ovnkube-node-bjt99","kube-system/konnectivity-agent-h5xkl","openshift-image-registry/image-registry-7fc9466475-jz9vh","openshift-service-ca/service-ca-5fffc97bdf-4cfcr","openshift-network-diagnostics/network-check-target-spx9v","openshift-insights/periodic-gathering-4sgw4-zcs8v","openshift-dns/node-resolver-9xqln","openshift-image-registry/node-ca-hnmxk","openshift-multus/network-metrics-daemon-bxbq6","openshift-ingress-canary/ingress-canary-ljjfm","openshift-image-registry/image-registry-74b44b76d6-l7p76","openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-956lw","openshift-network-operator/iptables-alerter-w9ckc","openshift-network-console/networking-console-plugin-86c478bb88-qx7zq","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fgpcv","kube-system/kas-connection-checker-685895c7c9-9m2lz","kube-system/global-pull-secret-syncer-2z8sh"]
Oct 30 21:33:26.801383 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.801362    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48"
Oct 30 21:33:26.802558 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.802539    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:26.803058 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.803037    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd"
Oct 30 21:33:26.807471 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.807448    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-qx7zq"
Oct 30 21:33:26.807471 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.807469    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:26.813060 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.813041    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:33:26.814173 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.814154    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:26.815418 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.815402    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:26.817177 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.817158    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:26.832400 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.832384    5087 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:33:26.832481 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:26.832433    5087 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:33:26.833034 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.833020    5087 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:33:26.833097 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:26.833062    5087 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-92.us-west-1.compute.internal"
Oct 30 21:33:26.853053 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.853031    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 30 21:33:26.853053 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.853032    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 30 21:33:26.853160 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.853137    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 30 21:33:26.853224 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.853211    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:33:26.855305 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.855287    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 30 21:33:26.855305 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.855311    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 30 21:33:26.855549 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.855341    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 30 21:33:26.855549 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.855368    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 21:33:26.855549 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.855411    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 21:33:26.855549 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.855533    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:33:26.855764 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.855551    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:33:26.855764 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.855560    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 21:33:26.855764 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.855622    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 30 21:33:26.855764 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.855690    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 30 21:33:26.855764 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.855709    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 30 21:33:26.855764 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.855733    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 21:33:26.856217 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.856202    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 30 21:33:26.860080 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.860065    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 30 21:33:26.860228 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.860214    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:33:26.860657 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.860645    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 30 21:33:26.862531 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.862513    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 30 21:33:26.862531 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.862517    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 30 21:33:26.862668 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.862551    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 30 21:33:26.862668 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.862556    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-hlnd6\""
Oct 30 21:33:26.862668 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.862642    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-8l7jw\""
Oct 30 21:33:26.862668 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.862644    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 30 21:33:26.862846 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.862753    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 30 21:33:26.862846 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.862783    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 30 21:33:26.862909 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.862875    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-vxknp\""
Oct 30 21:33:26.862948 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.862906    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-bj8tb\""
Oct 30 21:33:26.862993 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.862982    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 30 21:33:26.863030 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.862981    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-69h9r\""
Oct 30 21:33:26.863030 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.862981    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5vdw5\""
Oct 30 21:33:26.866920 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.866907    5087 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:33:26.878650 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.878632    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:33:26.878650 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.878639    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 21:33:26.878753 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.878647    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:33:26.878784 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.878770    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:33:26.878910 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.878898    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 30 21:33:26.879586 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.879570    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:33:26.879694 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.879681    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:33:26.879802 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.879789    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:33:26.880072 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.880035    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:33:26.880072 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.880056    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:33:26.880187 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.880087    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:33:26.880187 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.880123    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:33:26.880187 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.880168    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:33:26.880187 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.880175    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 30 21:33:26.880369 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.880176    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:33:26.880369 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.880202    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:33:26.880369 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.880231    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:33:26.880369 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.880225    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:33:26.880369 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.880296    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:33:26.880369 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.880307    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:33:26.880369 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.880309    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:33:26.880738 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.880433    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-qlt7z\""
Oct 30 21:33:26.880738 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.880441    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:33:26.880738 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.880453    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:33:26.880738 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.880569    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:33:26.880738 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.880596    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:33:26.880738 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.880621    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:33:26.880967 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.880783    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 21:33:26.880967 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.880795    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:33:26.880967 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.880819    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tvlkp\""
Oct 30 21:33:26.880967 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.880840    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:33:26.881093 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.881054    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 21:33:26.881129 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.881115    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 21:33:26.904986 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.904937    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:33:26.911632 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.911615    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-hlsb4\""
Oct 30 21:33:26.911854 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.911840    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 21:33:26.911890 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.911875    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 30 21:33:26.912021 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.912001    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:33:26.912433 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.912409    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-f9hfm\""
Oct 30 21:33:26.912433 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.912417    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:33:26.912555 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.912434    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6t698\""
Oct 30 21:33:26.912555 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.912449    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:33:26.912555 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.912441    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-wn5fv\""
Oct 30 21:33:26.912555 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.912437    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jnpcj\""
Oct 30 21:33:26.912735 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.912715    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rjmnn\""
Oct 30 21:33:26.912823 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.912813    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 30 21:33:26.928044 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.928018    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mzgn4\""
Oct 30 21:33:26.928122 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.928097    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-lbvv2\""
Oct 30 21:33:26.928297 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.928277    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 30 21:33:26.928297 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.928288    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-mbbvd\""
Oct 30 21:33:26.928412 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.928298    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9vgdc\""
Oct 30 21:33:26.928412 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.928340    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-cprz9\""
Oct 30 21:33:26.928591 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.928574    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-zs2mb\""
Oct 30 21:33:26.929423 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.929384    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 30 21:33:26.930269 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.930249    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2v7jg\""
Oct 30 21:33:26.930269 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.930257    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-76thg\""
Oct 30 21:33:26.930403 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.930265    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:33:26.930441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.930408    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:33:26.930441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.930408    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:33:26.930622 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.930588    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-26jx5\""
Oct 30 21:33:26.930622 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.930589    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-cftmq\""
Oct 30 21:33:26.930622 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.930618    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 21:33:26.930891 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.930631    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:33:26.930891 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.930653    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 21:33:26.930891 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.930678    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-9nw4b\""
Oct 30 21:33:26.930891 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.930683    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:33:26.930891 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.930702    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 21:33:26.930891 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.930741    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:33:26.930891 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.930769    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:33:26.930891 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.930696    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pplxq\""
Oct 30 21:33:26.930891 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.930792    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-flltm\""
Oct 30 21:33:26.930891 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.930799    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:33:26.930891 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.930802    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:33:26.930891 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.930745    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:33:26.930891 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.930821    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:33:26.930891 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.930825    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pczpt\""
Oct 30 21:33:26.930891 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.930810    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 21:33:26.930891 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.930848    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 21:33:26.930891 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.930866    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 21:33:26.931469 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.930906    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 21:33:26.931469 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.930999    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:33:26.957826 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.957804    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42dc1fb7-b6e0-40d2-bf53-20079815b12d-image-registry-private-configuration\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:26.957940 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.957835    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-socket-dir\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:33:26.957940 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.957853    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6chm6\" (UniqueName: \"kubernetes.io/projected/579a0c8f-9497-4448-98ab-4686adec22cd-kube-api-access-6chm6\") pod \"network-check-source-6f78d59f99-64b9m\" (UID: \"579a0c8f-9497-4448-98ab-4686adec22cd\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-64b9m"
Oct 30 21:33:26.957940 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.957872    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f54f9754-2024-41b9-8f84-fc3d02a22253-service-ca-bundle\") pod \"insights-operator-7c4986d449-z4jbc\" (UID: \"f54f9754-2024-41b9-8f84-fc3d02a22253\") " pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:26.957940 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.957898    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kgrcd\" (UID: \"f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd"
Oct 30 21:33:26.958157 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.957934    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7986120b-482f-4b0b-a605-32770569962d-ovnkube-config\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:26.958157 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.957982    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6gq8\" (UniqueName: \"kubernetes.io/projected/3d0811a7-4a83-4758-8ca0-2ac7dd5a9408-kube-api-access-h6gq8\") pod \"migrator-66df5d496b-fjrtf\" (UID: \"3d0811a7-4a83-4758-8ca0-2ac7dd5a9408\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fjrtf"
Oct 30 21:33:26.958157 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.958009    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-var-lib-kubelet\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:26.958157 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.958084    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcz4j\" (UniqueName: \"kubernetes.io/projected/e1a448f1-e180-4fbf-9923-fac0b9dbec74-kube-api-access-xcz4j\") pod \"network-check-target-spx9v\" (UID: \"e1a448f1-e180-4fbf-9923-fac0b9dbec74\") " pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:33:26.958157 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.958094    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f54f9754-2024-41b9-8f84-fc3d02a22253-service-ca-bundle\") pod \"insights-operator-7c4986d449-z4jbc\" (UID: \"f54f9754-2024-41b9-8f84-fc3d02a22253\") " pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:26.958157 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.958112    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42dc1fb7-b6e0-40d2-bf53-20079815b12d-image-registry-private-configuration\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:26.958157 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.958144    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-config-volume\") pod \"dns-default-5kfvz\" (UID: \"2a216b8f-a750-4e1e-9ab1-ff6c89d416e5\") " pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:26.958481 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.958188    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a724a1c0-c643-49ff-a555-5bddec4bfacb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:26.958481 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.958254    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-tls\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:26.958481 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.958321    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfn4f\" (UniqueName: \"kubernetes.io/projected/2a7ef1f3-4e9d-40db-93a3-4d38a3474e89-kube-api-access-kfn4f\") pod \"iptables-alerter-w9ckc\" (UID: \"2a7ef1f3-4e9d-40db-93a3-4d38a3474e89\") " pod="openshift-network-operator/iptables-alerter-w9ckc"
Oct 30 21:33:26.958481 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.958402    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7986120b-482f-4b0b-a605-32770569962d-ovnkube-config\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:26.958481 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.958401    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-etc-selinux\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:33:26.958481 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.958458    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-config-volume\") pod \"dns-default-5kfvz\" (UID: \"2a216b8f-a750-4e1e-9ab1-ff6c89d416e5\") " pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:26.958481 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.958459    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqsx2\" (UniqueName: \"kubernetes.io/projected/3ca424b7-c125-43ce-b837-88d1c02a05a0-kube-api-access-jqsx2\") pod \"volume-data-source-validator-58c9f5f46d-fgpcv\" (UID: \"3ca424b7-c125-43ce-b837-88d1c02a05a0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fgpcv"
Oct 30 21:33:26.958722 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.958489    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef-config\") pod \"service-ca-operator-544d749c8c-bn8fx\" (UID: \"f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bn8fx"
Oct 30 21:33:26.958722 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.958573    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs\") pod \"network-metrics-daemon-bxbq6\" (UID: \"032b9335-657e-4a86-aa6d-4ef46163014a\") " pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:33:26.958722 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.958599    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-etc-openvswitch\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:26.958722 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.958631    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a20999e6-4f67-47b9-9cf1-da1b1fb027f2-serviceca\") pod \"node-ca-hnmxk\" (UID: \"a20999e6-4f67-47b9-9cf1-da1b1fb027f2\") " pod="openshift-image-registry/node-ca-hnmxk"
Oct 30 21:33:26.958722 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.958674    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef-config\") pod \"service-ca-operator-544d749c8c-bn8fx\" (UID: \"f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bn8fx"
Oct 30 21:33:26.958890 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.958764    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc\" (UID: \"25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc"
Oct 30 21:33:26.958890 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.958812    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f54f9754-2024-41b9-8f84-fc3d02a22253-tmp\") pod \"insights-operator-7c4986d449-z4jbc\" (UID: \"f54f9754-2024-41b9-8f84-fc3d02a22253\") " pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:26.958890 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.958820    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a20999e6-4f67-47b9-9cf1-da1b1fb027f2-serviceca\") pod \"node-ca-hnmxk\" (UID: \"a20999e6-4f67-47b9-9cf1-da1b1fb027f2\") " pod="openshift-image-registry/node-ca-hnmxk"
Oct 30 21:33:26.959066 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.958911    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm65m\" (UniqueName: \"kubernetes.io/projected/f54f9754-2024-41b9-8f84-fc3d02a22253-kube-api-access-tm65m\") pod \"insights-operator-7c4986d449-z4jbc\" (UID: \"f54f9754-2024-41b9-8f84-fc3d02a22253\") " pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:26.959066 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.958932    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc\" (UID: \"25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc"
Oct 30 21:33:26.959066 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.958952    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f54f9754-2024-41b9-8f84-fc3d02a22253-tmp\") pod \"insights-operator-7c4986d449-z4jbc\" (UID: \"f54f9754-2024-41b9-8f84-fc3d02a22253\") " pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:26.959066 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.958960    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a724a1c0-c643-49ff-a555-5bddec4bfacb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:26.959191 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.959062    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-metrics-certs\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:26.959191 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.959106    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1196dab1-80d7-48eb-8d05-1a26ab6d0759-config\") pod \"console-operator-7cd56b8dbc-z8clf\" (UID: \"1196dab1-80d7-48eb-8d05-1a26ab6d0759\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf"
Oct 30 21:33:26.959191 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.959130    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whb6q\" (UniqueName: \"kubernetes.io/projected/a724a1c0-c643-49ff-a555-5bddec4bfacb-kube-api-access-whb6q\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:26.959341 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.959203    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2qz48\" (UID: \"9f16bd1f-a3bd-45ee-a02e-cb6b9e982327\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48"
Oct 30 21:33:26.959341 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.959235    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk9wl\" (UniqueName: \"kubernetes.io/projected/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-kube-api-access-hk9wl\") pod \"cluster-monitoring-operator-884576c57-2qz48\" (UID: \"9f16bd1f-a3bd-45ee-a02e-cb6b9e982327\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48"
Oct 30 21:33:26.959341 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.959264    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-var-lib-openvswitch\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:26.959341 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.959297    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1196dab1-80d7-48eb-8d05-1a26ab6d0759-config\") pod \"console-operator-7cd56b8dbc-z8clf\" (UID: \"1196dab1-80d7-48eb-8d05-1a26ab6d0759\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf"
Oct 30 21:33:26.959533 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.959384    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t68x5\" (UniqueName: \"kubernetes.io/projected/239a77e8-635c-4388-ad79-1cb3b15129d8-kube-api-access-t68x5\") pod \"node-resolver-9xqln\" (UID: \"239a77e8-635c-4388-ad79-1cb3b15129d8\") " pod="openshift-dns/node-resolver-9xqln"
Oct 30 21:33:26.959533 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.959410    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8z8h\" (UniqueName: \"kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-kube-api-access-f8z8h\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:26.959533 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.959463    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-run-openvswitch\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:26.959533 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.959488    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-kubernetes\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:26.959696 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.959564    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-var-lib-kubelet\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:26.959696 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.959589    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1196dab1-80d7-48eb-8d05-1a26ab6d0759-trusted-ca\") pod \"console-operator-7cd56b8dbc-z8clf\" (UID: \"1196dab1-80d7-48eb-8d05-1a26ab6d0759\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf"
Oct 30 21:33:26.959696 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.959622    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkjl7\" (UniqueName: \"kubernetes.io/projected/a20999e6-4f67-47b9-9cf1-da1b1fb027f2-kube-api-access-lkjl7\") pod \"node-ca-hnmxk\" (UID: \"a20999e6-4f67-47b9-9cf1-da1b1fb027f2\") " pod="openshift-image-registry/node-ca-hnmxk"
Oct 30 21:33:26.959839 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.959726    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc\" (UID: \"25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc"
Oct 30 21:33:26.959891 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.959834    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq5j2\" (UniqueName: \"kubernetes.io/projected/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-kube-api-access-rq5j2\") pod \"dns-default-5kfvz\" (UID: \"2a216b8f-a750-4e1e-9ab1-ff6c89d416e5\") " pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:26.959891 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.959866    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-kubelet\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:26.959987 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.959927    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc\" (UID: \"25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc"
Oct 30 21:33:26.960042 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.959985    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-metrics-tls\") pod \"dns-default-5kfvz\" (UID: \"2a216b8f-a750-4e1e-9ab1-ff6c89d416e5\") " pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:26.960042 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960009    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-run-ovn\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:26.960042 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960038    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-modprobe-d\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:26.960183 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960063    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/752a05d6-f630-42ec-9795-0d28e9ed7d80-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:33:26.960183 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960096    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-run-ovn-kubernetes\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:26.960183 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960153    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:26.960331 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960192    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwsj7\" (UniqueName: \"kubernetes.io/projected/752a05d6-f630-42ec-9795-0d28e9ed7d80-kube-api-access-qwsj7\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:33:26.960331 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960239    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckxzq\" (UniqueName: \"kubernetes.io/projected/298f2a29-a322-4977-bd9f-d6b765eff5d1-kube-api-access-ckxzq\") pod \"migrator-66df5d496b-mf2wv\" (UID: \"298f2a29-a322-4977-bd9f-d6b765eff5d1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mf2wv"
Oct 30 21:33:26.960331 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960263    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2qz48\" (UID: \"9f16bd1f-a3bd-45ee-a02e-cb6b9e982327\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48"
Oct 30 21:33:26.960331 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960294    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-host\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:26.960553 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960394    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/061313cb-4d50-4ae5-afda-dc63a62a013b-tmp\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:26.960553 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960413    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/752a05d6-f630-42ec-9795-0d28e9ed7d80-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:33:26.960553 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960427    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9a1460f5-20f5-4bd1-843f-3838935f2ce2-multus-daemon-config\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:26.960553 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960452    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/752a05d6-f630-42ec-9795-0d28e9ed7d80-cnibin\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:33:26.960553 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960477    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42dc1fb7-b6e0-40d2-bf53-20079815b12d-trusted-ca\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:26.960553 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960492    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-kubelet-config\") pod \"global-pull-secret-syncer-2z8sh\" (UID: \"2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49\") " pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:33:26.960553 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960490    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/061313cb-4d50-4ae5-afda-dc63a62a013b-tmp\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:26.960553 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960529    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2qz48\" (UID: \"9f16bd1f-a3bd-45ee-a02e-cb6b9e982327\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48"
Oct 30 21:33:26.960553 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960554    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-kgrcd\" (UID: \"f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd"
Oct 30 21:33:26.961003 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960578    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2mp5\" (UniqueName: \"kubernetes.io/projected/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-kube-api-access-p2mp5\") pod \"cluster-samples-operator-6ddb6c8b94-kgrcd\" (UID: \"f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd"
Oct 30 21:33:26.961003 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960595    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/98a3aaba-8c58-43ae-837d-a809085d692c-signing-cabundle\") pod \"service-ca-5fffc97bdf-4cfcr\" (UID: \"98a3aaba-8c58-43ae-837d-a809085d692c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4cfcr"
Oct 30 21:33:26.961003 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960615    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-sysctl-d\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:26.961003 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960639    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/752a05d6-f630-42ec-9795-0d28e9ed7d80-tuning-conf-dir\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:33:26.961003 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960662    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-slash\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:26.961003 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960662    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9a1460f5-20f5-4bd1-843f-3838935f2ce2-multus-daemon-config\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:26.961003 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960682    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-log-socket\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:26.961003 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960733    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-run\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:26.961003 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960760    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-kgrcd\" (UID: \"f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd"
Oct 30 21:33:26.961003 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960762    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2a7ef1f3-4e9d-40db-93a3-4d38a3474e89-iptables-alerter-script\") pod \"iptables-alerter-w9ckc\" (UID: \"2a7ef1f3-4e9d-40db-93a3-4d38a3474e89\") " pod="openshift-network-operator/iptables-alerter-w9ckc"
Oct 30 21:33:26.961003 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960840    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/98a3aaba-8c58-43ae-837d-a809085d692c-signing-cabundle\") pod \"service-ca-5fffc97bdf-4cfcr\" (UID: \"98a3aaba-8c58-43ae-837d-a809085d692c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4cfcr"
Oct 30 21:33:26.961003 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960861    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-run-k8s-cni-cncf-io\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:26.961003 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960905    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-hostroot\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:26.961003 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960936    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5278b436-9321-4856-b9b0-ce9eb2596775-service-ca-bundle\") pod \"periodic-gathering-4sgw4-zcs8v\" (UID: \"5278b436-9321-4856-b9b0-ce9eb2596775\") " pod="openshift-insights/periodic-gathering-4sgw4-zcs8v"
Oct 30 21:33:26.961003 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960966    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gw68\" (UniqueName: \"kubernetes.io/projected/b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5-kube-api-access-4gw68\") pod \"ingress-canary-ljjfm\" (UID: \"b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5\") " pod="openshift-ingress-canary/ingress-canary-ljjfm"
Oct 30 21:33:26.961003 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.960984    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2a7ef1f3-4e9d-40db-93a3-4d38a3474e89-iptables-alerter-script\") pod \"iptables-alerter-w9ckc\" (UID: \"2a7ef1f3-4e9d-40db-93a3-4d38a3474e89\") " pod="openshift-network-operator/iptables-alerter-w9ckc"
Oct 30 21:33:26.961003 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961004    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-stats-auth\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961025    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7986120b-482f-4b0b-a605-32770569962d-ovn-node-metrics-cert\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961045    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d5f25065-34c9-4724-8dff-9ace0890d565-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qx7zq\" (UID: \"d5f25065-34c9-4724-8dff-9ace0890d565\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qx7zq"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961062    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-tls\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961080    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42dc1fb7-b6e0-40d2-bf53-20079815b12d-installation-pull-secrets\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961096    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/239a77e8-635c-4388-ad79-1cb3b15129d8-hosts-file\") pod \"node-resolver-9xqln\" (UID: \"239a77e8-635c-4388-ad79-1cb3b15129d8\") " pod="openshift-dns/node-resolver-9xqln"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961094    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1196dab1-80d7-48eb-8d05-1a26ab6d0759-trusted-ca\") pod \"console-operator-7cd56b8dbc-z8clf\" (UID: \"1196dab1-80d7-48eb-8d05-1a26ab6d0759\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961112    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-certificates\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961118    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5278b436-9321-4856-b9b0-ce9eb2596775-service-ca-bundle\") pod \"periodic-gathering-4sgw4-zcs8v\" (UID: \"5278b436-9321-4856-b9b0-ce9eb2596775\") " pod="openshift-insights/periodic-gathering-4sgw4-zcs8v"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961148    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-certificates\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961163    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42dc1fb7-b6e0-40d2-bf53-20079815b12d-trusted-ca\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961181    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a724a1c0-c643-49ff-a555-5bddec4bfacb-crio-socket\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961212    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-run-netns\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961224    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-certificates\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961237    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-tuned\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961257    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-certificates\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961266    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5278b436-9321-4856-b9b0-ce9eb2596775-archives-path\") pod \"periodic-gathering-4sgw4-zcs8v\" (UID: \"5278b436-9321-4856-b9b0-ce9eb2596775\") " pod="openshift-insights/periodic-gathering-4sgw4-zcs8v"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961291    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrr6v\" (UniqueName: \"kubernetes.io/projected/1196dab1-80d7-48eb-8d05-1a26ab6d0759-kube-api-access-qrr6v\") pod \"console-operator-7cd56b8dbc-z8clf\" (UID: \"1196dab1-80d7-48eb-8d05-1a26ab6d0759\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961308    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48g5f\" (UniqueName: \"kubernetes.io/projected/5278b436-9321-4856-b9b0-ce9eb2596775-kube-api-access-48g5f\") pod \"periodic-gathering-4sgw4-zcs8v\" (UID: \"5278b436-9321-4856-b9b0-ce9eb2596775\") " pod="openshift-insights/periodic-gathering-4sgw4-zcs8v"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961317    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-tuned\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961325    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/239a77e8-635c-4388-ad79-1cb3b15129d8-tmp-dir\") pod \"node-resolver-9xqln\" (UID: \"239a77e8-635c-4388-ad79-1cb3b15129d8\") " pod="openshift-dns/node-resolver-9xqln"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961379    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5278b436-9321-4856-b9b0-ce9eb2596775-archives-path\") pod \"periodic-gathering-4sgw4-zcs8v\" (UID: \"5278b436-9321-4856-b9b0-ce9eb2596775\") " pod="openshift-insights/periodic-gathering-4sgw4-zcs8v"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961393    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-stats-auth\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961426    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/239a77e8-635c-4388-ad79-1cb3b15129d8-tmp-dir\") pod \"node-resolver-9xqln\" (UID: \"239a77e8-635c-4388-ad79-1cb3b15129d8\") " pod="openshift-dns/node-resolver-9xqln"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961439    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-tmp-dir\") pod \"dns-default-5kfvz\" (UID: \"2a216b8f-a750-4e1e-9ab1-ff6c89d416e5\") " pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961495    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42dc1fb7-b6e0-40d2-bf53-20079815b12d-installation-pull-secrets\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961528    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a7ef1f3-4e9d-40db-93a3-4d38a3474e89-host-slash\") pod \"iptables-alerter-w9ckc\" (UID: \"2a7ef1f3-4e9d-40db-93a3-4d38a3474e89\") " pod="openshift-network-operator/iptables-alerter-w9ckc"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961564    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-tmp-dir\") pod \"dns-default-5kfvz\" (UID: \"2a216b8f-a750-4e1e-9ab1-ff6c89d416e5\") " pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961606    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7986120b-482f-4b0b-a605-32770569962d-ovn-node-metrics-cert\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961608    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5278b436-9321-4856-b9b0-ce9eb2596775-trusted-ca-bundle\") pod \"periodic-gathering-4sgw4-zcs8v\" (UID: \"5278b436-9321-4856-b9b0-ce9eb2596775\") " pod="openshift-insights/periodic-gathering-4sgw4-zcs8v"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961655    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f54f9754-2024-41b9-8f84-fc3d02a22253-snapshots\") pod \"insights-operator-7c4986d449-z4jbc\" (UID: \"f54f9754-2024-41b9-8f84-fc3d02a22253\") " pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961680    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f54f9754-2024-41b9-8f84-fc3d02a22253-serving-cert\") pod \"insights-operator-7c4986d449-z4jbc\" (UID: \"f54f9754-2024-41b9-8f84-fc3d02a22253\") " pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961705    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/98a3aaba-8c58-43ae-837d-a809085d692c-signing-key\") pod \"service-ca-5fffc97bdf-4cfcr\" (UID: \"98a3aaba-8c58-43ae-837d-a809085d692c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4cfcr"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961729    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/752a05d6-f630-42ec-9795-0d28e9ed7d80-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961736    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f54f9754-2024-41b9-8f84-fc3d02a22253-snapshots\") pod \"insights-operator-7c4986d449-z4jbc\" (UID: \"f54f9754-2024-41b9-8f84-fc3d02a22253\") " pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961755    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-lib-modules\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961780    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdkhn\" (UniqueName: \"kubernetes.io/projected/061313cb-4d50-4ae5-afda-dc63a62a013b-kube-api-access-qdkhn\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961811    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28w48\" (UniqueName: \"kubernetes.io/projected/8a74b249-57da-4405-af8e-ae777a44415e-kube-api-access-28w48\") pod \"kas-connection-checker-685895c7c9-9m2lz\" (UID: \"8a74b249-57da-4405-af8e-ae777a44415e\") " pod="kube-system/kas-connection-checker-685895c7c9-9m2lz"
Oct 30 21:33:26.961811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961861    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-run-multus-certs\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961922    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjbfn\" (UniqueName: \"kubernetes.io/projected/032b9335-657e-4a86-aa6d-4ef46163014a-kube-api-access-xjbfn\") pod \"network-metrics-daemon-bxbq6\" (UID: \"032b9335-657e-4a86-aa6d-4ef46163014a\") " pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961945    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/752a05d6-f630-42ec-9795-0d28e9ed7d80-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961960    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5278b436-9321-4856-b9b0-ce9eb2596775-trusted-ca-bundle\") pod \"periodic-gathering-4sgw4-zcs8v\" (UID: \"5278b436-9321-4856-b9b0-ce9eb2596775\") " pod="openshift-insights/periodic-gathering-4sgw4-zcs8v"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961952    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7986120b-482f-4b0b-a605-32770569962d-ovnkube-script-lib\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.961962    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/98a3aaba-8c58-43ae-837d-a809085d692c-signing-key\") pod \"service-ca-5fffc97bdf-4cfcr\" (UID: \"98a3aaba-8c58-43ae-837d-a809085d692c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4cfcr"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962091    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f54f9754-2024-41b9-8f84-fc3d02a22253-serving-cert\") pod \"insights-operator-7c4986d449-z4jbc\" (UID: \"f54f9754-2024-41b9-8f84-fc3d02a22253\") " pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962093    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42dc1fb7-b6e0-40d2-bf53-20079815b12d-ca-trust-extracted\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962155    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42dc1fb7-b6e0-40d2-bf53-20079815b12d-ca-trust-extracted\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962163    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83917e8e-50b6-4c52-b337-6f1be5a2c26a-service-ca-bundle\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962190    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/392cc7ef-7cd2-4090-a246-0347d5af7bbf-ca-trust-extracted\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962158    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7986120b-482f-4b0b-a605-32770569962d-ovnkube-script-lib\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962226    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgwnq\" (UniqueName: \"kubernetes.io/projected/7986120b-482f-4b0b-a605-32770569962d-kube-api-access-pgwnq\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962251    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-systemd\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962275    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-cnibin\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962299    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-run-netns\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962305    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/392cc7ef-7cd2-4090-a246-0347d5af7bbf-ca-trust-extracted\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962335    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f54f9754-2024-41b9-8f84-fc3d02a22253-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-z4jbc\" (UID: \"f54f9754-2024-41b9-8f84-fc3d02a22253\") " pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962380    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57sgv\" (UniqueName: \"kubernetes.io/projected/f84984eb-c144-4394-bb77-afc0c94b3a3a-kube-api-access-57sgv\") pod \"kas-connection-checker-685895c7c9-cx7d4\" (UID: \"f84984eb-c144-4394-bb77-afc0c94b3a3a\") " pod="kube-system/kas-connection-checker-685895c7c9-cx7d4"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962403    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-sysconfig\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962421    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkvjz\" (UniqueName: \"kubernetes.io/projected/9a1460f5-20f5-4bd1-843f-3838935f2ce2-kube-api-access-lkvjz\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962454    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-registration-dir\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962585    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77rbg\" (UniqueName: \"kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-kube-api-access-77rbg\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962606    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/392cc7ef-7cd2-4090-a246-0347d5af7bbf-image-registry-private-configuration\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962626    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-system-cni-dir\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962642    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/752a05d6-f630-42ec-9795-0d28e9ed7d80-system-cni-dir\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962659    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-multus-cni-dir\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962695    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f54f9754-2024-41b9-8f84-fc3d02a22253-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-z4jbc\" (UID: \"f54f9754-2024-41b9-8f84-fc3d02a22253\") " pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962695    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-multus-conf-dir\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962730    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962752    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-device-dir\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962789    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rbpk\" (UniqueName: \"kubernetes.io/projected/23a040b8-2420-4d59-b426-f52832dc7de1-kube-api-access-7rbpk\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962817    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/392cc7ef-7cd2-4090-a246-0347d5af7bbf-image-registry-private-configuration\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962837    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef-serving-cert\") pod \"service-ca-operator-544d749c8c-bn8fx\" (UID: \"f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bn8fx"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962879    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbqkr\" (UniqueName: \"kubernetes.io/projected/f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef-kube-api-access-fbqkr\") pod \"service-ca-operator-544d749c8c-bn8fx\" (UID: \"f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bn8fx"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962961    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a1460f5-20f5-4bd1-843f-3838935f2ce2-cni-binary-copy\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.962986    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef-serving-cert\") pod \"service-ca-operator-544d749c8c-bn8fx\" (UID: \"f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bn8fx"
Oct 30 21:33:26.963038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963035    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d5f25065-34c9-4724-8dff-9ace0890d565-nginx-conf\") pod \"networking-console-plugin-86c478bb88-qx7zq\" (UID: \"d5f25065-34c9-4724-8dff-9ace0890d565\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qx7zq"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963076    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/752a05d6-f630-42ec-9795-0d28e9ed7d80-cni-binary-copy\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963134    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a1460f5-20f5-4bd1-843f-3838935f2ce2-cni-binary-copy\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963164    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fbj2\" (UniqueName: \"kubernetes.io/projected/98a3aaba-8c58-43ae-837d-a809085d692c-kube-api-access-4fbj2\") pod \"service-ca-5fffc97bdf-4cfcr\" (UID: \"98a3aaba-8c58-43ae-837d-a809085d692c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4cfcr"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963206    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d5f25065-34c9-4724-8dff-9ace0890d565-nginx-conf\") pod \"networking-console-plugin-86c478bb88-qx7zq\" (UID: \"d5f25065-34c9-4724-8dff-9ace0890d565\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qx7zq"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963262    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/392cc7ef-7cd2-4090-a246-0347d5af7bbf-trusted-ca\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963285    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/752a05d6-f630-42ec-9795-0d28e9ed7d80-cni-binary-copy\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963282    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/392cc7ef-7cd2-4090-a246-0347d5af7bbf-installation-pull-secrets\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963315    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-bound-sa-token\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963363    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7986120b-482f-4b0b-a605-32770569962d-env-overrides\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963381    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5-cert\") pod \"ingress-canary-ljjfm\" (UID: \"b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5\") " pod="openshift-ingress-canary/ingress-canary-ljjfm"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963396    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-default-certificate\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963417    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-multus-socket-dir-parent\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963433    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a724a1c0-c643-49ff-a555-5bddec4bfacb-data-volume\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963453    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6e5648ef-56ca-4d99-ad1a-0adb0fc50855-agent-certs\") pod \"konnectivity-agent-h5xkl\" (UID: \"6e5648ef-56ca-4d99-ad1a-0adb0fc50855\") " pod="kube-system/konnectivity-agent-h5xkl"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963484    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-systemd-units\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963510    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/392cc7ef-7cd2-4090-a246-0347d5af7bbf-installation-pull-secrets\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963561    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a724a1c0-c643-49ff-a555-5bddec4bfacb-data-volume\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963590    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/392cc7ef-7cd2-4090-a246-0347d5af7bbf-trusted-ca\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963639    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-default-certificate\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963661    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7986120b-482f-4b0b-a605-32770569962d-env-overrides\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963673    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzjqd\" (UniqueName: \"kubernetes.io/projected/364111d6-7289-47be-8cf0-c10c7b6fbcdc-kube-api-access-lzjqd\") pod \"kas-connection-checker-685895c7c9-9trcs\" (UID: \"364111d6-7289-47be-8cf0-c10c7b6fbcdc\") " pod="kube-system/kas-connection-checker-685895c7c9-9trcs"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963707    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-bound-sa-token\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963735    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-dbus\") pod \"global-pull-secret-syncer-2z8sh\" (UID: \"2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49\") " pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963760    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-run-systemd\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963737    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5-cert\") pod \"ingress-canary-ljjfm\" (UID: \"b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5\") " pod="openshift-ingress-canary/ingress-canary-ljjfm"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963776    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-node-log\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963791    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-etc-kubernetes\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963824    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6e5648ef-56ca-4d99-ad1a-0adb0fc50855-agent-certs\") pod \"konnectivity-agent-h5xkl\" (UID: \"6e5648ef-56ca-4d99-ad1a-0adb0fc50855\") " pod="kube-system/konnectivity-agent-h5xkl"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963829    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-var-lib-cni-multus\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963847    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1196dab1-80d7-48eb-8d05-1a26ab6d0759-serving-cert\") pod \"console-operator-7cd56b8dbc-z8clf\" (UID: \"1196dab1-80d7-48eb-8d05-1a26ab6d0759\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963874    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a20999e6-4f67-47b9-9cf1-da1b1fb027f2-host\") pod \"node-ca-hnmxk\" (UID: \"a20999e6-4f67-47b9-9cf1-da1b1fb027f2\") " pod="openshift-image-registry/node-ca-hnmxk"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963904    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-original-pull-secret\") pod \"global-pull-secret-syncer-2z8sh\" (UID: \"2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49\") " pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963927    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6e5648ef-56ca-4d99-ad1a-0adb0fc50855-konnectivity-ca\") pod \"konnectivity-agent-h5xkl\" (UID: \"6e5648ef-56ca-4d99-ad1a-0adb0fc50855\") " pod="kube-system/konnectivity-agent-h5xkl"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963943    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-cni-netd\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963958    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-sysctl-conf\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.963976    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-sys\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.964033    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgn2z\" (UniqueName: \"kubernetes.io/projected/25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245-kube-api-access-tgn2z\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc\" (UID: \"25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.964067    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-sys-fs\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.964106    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wd6v\" (UniqueName: \"kubernetes.io/projected/83917e8e-50b6-4c52-b337-6f1be5a2c26a-kube-api-access-9wd6v\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.964124    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-original-pull-secret\") pod \"global-pull-secret-syncer-2z8sh\" (UID: \"2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49\") " pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.964129    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6e5648ef-56ca-4d99-ad1a-0adb0fc50855-konnectivity-ca\") pod \"konnectivity-agent-h5xkl\" (UID: \"6e5648ef-56ca-4d99-ad1a-0adb0fc50855\") " pod="kube-system/konnectivity-agent-h5xkl"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.964134    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-cni-bin\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.964181    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-var-lib-cni-bin\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:26.964391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:26.964207    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1196dab1-80d7-48eb-8d05-1a26ab6d0759-serving-cert\") pod \"console-operator-7cd56b8dbc-z8clf\" (UID: \"1196dab1-80d7-48eb-8d05-1a26ab6d0759\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf"
Oct 30 21:33:27.014638 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.014614    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqsx2\" (UniqueName: \"kubernetes.io/projected/3ca424b7-c125-43ce-b837-88d1c02a05a0-kube-api-access-jqsx2\") pod \"volume-data-source-validator-58c9f5f46d-fgpcv\" (UID: \"3ca424b7-c125-43ce-b837-88d1c02a05a0\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-fgpcv"
Oct 30 21:33:27.014797 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.014780    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6gq8\" (UniqueName: \"kubernetes.io/projected/3d0811a7-4a83-4758-8ca0-2ac7dd5a9408-kube-api-access-h6gq8\") pod \"migrator-66df5d496b-fjrtf\" (UID: \"3d0811a7-4a83-4758-8ca0-2ac7dd5a9408\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-fjrtf"
Oct 30 21:33:27.015101 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.015081    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfn4f\" (UniqueName: \"kubernetes.io/projected/2a7ef1f3-4e9d-40db-93a3-4d38a3474e89-kube-api-access-kfn4f\") pod \"iptables-alerter-w9ckc\" (UID: \"2a7ef1f3-4e9d-40db-93a3-4d38a3474e89\") " pod="openshift-network-operator/iptables-alerter-w9ckc"
Oct 30 21:33:27.016255 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.016231    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whb6q\" (UniqueName: \"kubernetes.io/projected/a724a1c0-c643-49ff-a555-5bddec4bfacb-kube-api-access-whb6q\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:27.016362 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.016284    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcz4j\" (UniqueName: \"kubernetes.io/projected/e1a448f1-e180-4fbf-9923-fac0b9dbec74-kube-api-access-xcz4j\") pod \"network-check-target-spx9v\" (UID: \"e1a448f1-e180-4fbf-9923-fac0b9dbec74\") " pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:33:27.016514 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.016495    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6chm6\" (UniqueName: \"kubernetes.io/projected/579a0c8f-9497-4448-98ab-4686adec22cd-kube-api-access-6chm6\") pod \"network-check-source-6f78d59f99-64b9m\" (UID: \"579a0c8f-9497-4448-98ab-4686adec22cd\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-64b9m"
Oct 30 21:33:27.016739 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.016719    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq5j2\" (UniqueName: \"kubernetes.io/projected/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-kube-api-access-rq5j2\") pod \"dns-default-5kfvz\" (UID: \"2a216b8f-a750-4e1e-9ab1-ff6c89d416e5\") " pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:27.016800 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.016773    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t68x5\" (UniqueName: \"kubernetes.io/projected/239a77e8-635c-4388-ad79-1cb3b15129d8-kube-api-access-t68x5\") pod \"node-resolver-9xqln\" (UID: \"239a77e8-635c-4388-ad79-1cb3b15129d8\") " pod="openshift-dns/node-resolver-9xqln"
Oct 30 21:33:27.016876 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.016856    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkjl7\" (UniqueName: \"kubernetes.io/projected/a20999e6-4f67-47b9-9cf1-da1b1fb027f2-kube-api-access-lkjl7\") pod \"node-ca-hnmxk\" (UID: \"a20999e6-4f67-47b9-9cf1-da1b1fb027f2\") " pod="openshift-image-registry/node-ca-hnmxk"
Oct 30 21:33:27.016909 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.016892    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8z8h\" (UniqueName: \"kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-kube-api-access-f8z8h\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:27.017209 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.017193    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm65m\" (UniqueName: \"kubernetes.io/projected/f54f9754-2024-41b9-8f84-fc3d02a22253-kube-api-access-tm65m\") pod \"insights-operator-7c4986d449-z4jbc\" (UID: \"f54f9754-2024-41b9-8f84-fc3d02a22253\") " pod="openshift-insights/insights-operator-7c4986d449-z4jbc"
Oct 30 21:33:27.017247 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.017216    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk9wl\" (UniqueName: \"kubernetes.io/projected/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-kube-api-access-hk9wl\") pod \"cluster-monitoring-operator-884576c57-2qz48\" (UID: \"9f16bd1f-a3bd-45ee-a02e-cb6b9e982327\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48"
Oct 30 21:33:27.018033 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.018015    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckxzq\" (UniqueName: \"kubernetes.io/projected/298f2a29-a322-4977-bd9f-d6b765eff5d1-kube-api-access-ckxzq\") pod \"migrator-66df5d496b-mf2wv\" (UID: \"298f2a29-a322-4977-bd9f-d6b765eff5d1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-mf2wv"
Oct 30 21:33:27.018195 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.018179    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwsj7\" (UniqueName: \"kubernetes.io/projected/752a05d6-f630-42ec-9795-0d28e9ed7d80-kube-api-access-qwsj7\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:33:27.018658 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.018639    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gw68\" (UniqueName: \"kubernetes.io/projected/b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5-kube-api-access-4gw68\") pod \"ingress-canary-ljjfm\" (UID: \"b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5\") " pod="openshift-ingress-canary/ingress-canary-ljjfm"
Oct 30 21:33:27.018802 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.018786    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2mp5\" (UniqueName: \"kubernetes.io/projected/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-kube-api-access-p2mp5\") pod \"cluster-samples-operator-6ddb6c8b94-kgrcd\" (UID: \"f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd"
Oct 30 21:33:27.022461 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.022440    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-bound-sa-token\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:27.022663 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.022648    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjbfn\" (UniqueName: \"kubernetes.io/projected/032b9335-657e-4a86-aa6d-4ef46163014a-kube-api-access-xjbfn\") pod \"network-metrics-daemon-bxbq6\" (UID: \"032b9335-657e-4a86-aa6d-4ef46163014a\") " pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:33:27.023126 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.023110    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzjqd\" (UniqueName: \"kubernetes.io/projected/364111d6-7289-47be-8cf0-c10c7b6fbcdc-kube-api-access-lzjqd\") pod \"kas-connection-checker-685895c7c9-9trcs\" (UID: \"364111d6-7289-47be-8cf0-c10c7b6fbcdc\") " pod="kube-system/kas-connection-checker-685895c7c9-9trcs"
Oct 30 21:33:27.023192 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.023179    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fbj2\" (UniqueName: \"kubernetes.io/projected/98a3aaba-8c58-43ae-837d-a809085d692c-kube-api-access-4fbj2\") pod \"service-ca-5fffc97bdf-4cfcr\" (UID: \"98a3aaba-8c58-43ae-837d-a809085d692c\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4cfcr"
Oct 30 21:33:27.023230 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.023216    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48g5f\" (UniqueName: \"kubernetes.io/projected/5278b436-9321-4856-b9b0-ce9eb2596775-kube-api-access-48g5f\") pod \"periodic-gathering-4sgw4-zcs8v\" (UID: \"5278b436-9321-4856-b9b0-ce9eb2596775\") " pod="openshift-insights/periodic-gathering-4sgw4-zcs8v"
Oct 30 21:33:27.023816 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.023800    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgn2z\" (UniqueName: \"kubernetes.io/projected/25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245-kube-api-access-tgn2z\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc\" (UID: \"25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc"
Oct 30 21:33:27.024241 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.024226    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28w48\" (UniqueName: \"kubernetes.io/projected/8a74b249-57da-4405-af8e-ae777a44415e-kube-api-access-28w48\") pod \"kas-connection-checker-685895c7c9-9m2lz\" (UID: \"8a74b249-57da-4405-af8e-ae777a44415e\") " pod="kube-system/kas-connection-checker-685895c7c9-9m2lz"
Oct 30 21:33:27.024384 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.024370    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rbpk\" (UniqueName: \"kubernetes.io/projected/23a040b8-2420-4d59-b426-f52832dc7de1-kube-api-access-7rbpk\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:33:27.024902 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.024887    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgwnq\" (UniqueName: \"kubernetes.io/projected/7986120b-482f-4b0b-a605-32770569962d-kube-api-access-pgwnq\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.025315 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.025298    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkvjz\" (UniqueName: \"kubernetes.io/projected/9a1460f5-20f5-4bd1-843f-3838935f2ce2-kube-api-access-lkvjz\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:27.025362 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.025311    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-bound-sa-token\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:27.025517 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.025501    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbqkr\" (UniqueName: \"kubernetes.io/projected/f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef-kube-api-access-fbqkr\") pod \"service-ca-operator-544d749c8c-bn8fx\" (UID: \"f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bn8fx"
Oct 30 21:33:27.026170 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.026152    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wd6v\" (UniqueName: \"kubernetes.io/projected/83917e8e-50b6-4c52-b337-6f1be5a2c26a-kube-api-access-9wd6v\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:27.026311 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.026292    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77rbg\" (UniqueName: \"kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-kube-api-access-77rbg\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:27.026474 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.026459    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57sgv\" (UniqueName: \"kubernetes.io/projected/f84984eb-c144-4394-bb77-afc0c94b3a3a-kube-api-access-57sgv\") pod \"kas-connection-checker-685895c7c9-cx7d4\" (UID: \"f84984eb-c144-4394-bb77-afc0c94b3a3a\") " pod="kube-system/kas-connection-checker-685895c7c9-cx7d4"
Oct 30 21:33:27.026515 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.026494    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdkhn\" (UniqueName: \"kubernetes.io/projected/061313cb-4d50-4ae5-afda-dc63a62a013b-kube-api-access-qdkhn\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:27.026578 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.026566    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrr6v\" (UniqueName: \"kubernetes.io/projected/1196dab1-80d7-48eb-8d05-1a26ab6d0759-kube-api-access-qrr6v\") pod \"console-operator-7cd56b8dbc-z8clf\" (UID: \"1196dab1-80d7-48eb-8d05-1a26ab6d0759\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf"
Oct 30 21:33:27.064663 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.064643    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-multus-socket-dir-parent\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:27.064768 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.064670    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-systemd-units\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.064768 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.064708    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-dbus\") pod \"global-pull-secret-syncer-2z8sh\" (UID: \"2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49\") " pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:33:27.064768 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.064728    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-run-systemd\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.064768 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.064750    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-node-log\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.064962 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.064766    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-systemd-units\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.064962 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.064764    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-multus-socket-dir-parent\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:27.064962 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.064790    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-run-systemd\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.064962 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.064797    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-node-log\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.064962 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.064797    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-dbus\") pod \"global-pull-secret-syncer-2z8sh\" (UID: \"2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49\") " pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:33:27.064962 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.064811    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-etc-kubernetes\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:27.064962 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.064838    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-var-lib-cni-multus\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:27.064962 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.064854    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a20999e6-4f67-47b9-9cf1-da1b1fb027f2-host\") pod \"node-ca-hnmxk\" (UID: \"a20999e6-4f67-47b9-9cf1-da1b1fb027f2\") " pod="openshift-image-registry/node-ca-hnmxk"
Oct 30 21:33:27.064962 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.064861    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-etc-kubernetes\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:27.064962 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.064870    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-cni-netd\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.064962 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.064893    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-sysctl-conf\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:27.064962 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.064909    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-sys\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:27.064962 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.064912    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-var-lib-cni-multus\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:27.064962 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.064923    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-cni-netd\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.064962 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.064924    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-sys-fs\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:33:27.064962 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.064958    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-sys-fs\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:33:27.064962 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.064964    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-cni-bin\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.064962 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.064908    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a20999e6-4f67-47b9-9cf1-da1b1fb027f2-host\") pod \"node-ca-hnmxk\" (UID: \"a20999e6-4f67-47b9-9cf1-da1b1fb027f2\") " pod="openshift-image-registry/node-ca-hnmxk"
Oct 30 21:33:27.064962 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.064971    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-sys\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.064988    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-var-lib-cni-bin\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065009    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-sysctl-conf\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065005    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-cni-bin\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065047    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-socket-dir\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065061    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-var-lib-cni-bin\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065104    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-socket-dir\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065123    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kgrcd\" (UID: \"f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065158    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-var-lib-kubelet\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:27.065189    5087 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065213    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-var-lib-kubelet\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065188    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a724a1c0-c643-49ff-a555-5bddec4bfacb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:27.065260    5087 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-tls podName:f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf nodeName:}" failed. No retries permitted until 2026-10-30 21:33:27.565232543 +0000 UTC m=+2.039736528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-kgrcd" (UID: "f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf") : secret "samples-operator-tls" not found
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:27.065262    5087 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065272    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-tls\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065288    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-etc-selinux\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065305    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs\") pod \"network-metrics-daemon-bxbq6\" (UID: \"032b9335-657e-4a86-aa6d-4ef46163014a\") " pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:27.065318    5087 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a724a1c0-c643-49ff-a555-5bddec4bfacb-insights-runtime-extractor-tls podName:a724a1c0-c643-49ff-a555-5bddec4bfacb nodeName:}" failed. No retries permitted until 2026-10-30 21:33:27.565302401 +0000 UTC m=+2.039806373 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a724a1c0-c643-49ff-a555-5bddec4bfacb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g27zs" (UID: "a724a1c0-c643-49ff-a555-5bddec4bfacb") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:27.065368    5087 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065370    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-etc-openvswitch\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065393    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-etc-selinux\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:27.065400    5087 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs podName:032b9335-657e-4a86-aa6d-4ef46163014a nodeName:}" failed. No retries permitted until 2026-10-30 21:33:27.565392007 +0000 UTC m=+2.039895963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs") pod "network-metrics-daemon-bxbq6" (UID: "032b9335-657e-4a86-aa6d-4ef46163014a") : secret "metrics-daemon-secret" not found
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065412    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a724a1c0-c643-49ff-a555-5bddec4bfacb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065425    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-etc-openvswitch\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065429    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-metrics-certs\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065465    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2qz48\" (UID: \"9f16bd1f-a3bd-45ee-a02e-cb6b9e982327\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065493    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-var-lib-openvswitch\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065535    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-var-lib-openvswitch\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:27.065541    5087 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:27.065572    5087 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls podName:9f16bd1f-a3bd-45ee-a02e-cb6b9e982327 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:27.56556517 +0000 UTC m=+2.040069174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2qz48" (UID: "9f16bd1f-a3bd-45ee-a02e-cb6b9e982327") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065567    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-run-openvswitch\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065592    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-kubernetes\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065599    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-run-openvswitch\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065607    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-var-lib-kubelet\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065638    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-kubernetes\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065651    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-var-lib-kubelet\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065647    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-kubelet\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065673    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-kubelet\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065676    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-metrics-tls\") pod \"dns-default-5kfvz\" (UID: \"2a216b8f-a750-4e1e-9ab1-ff6c89d416e5\") " pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065699    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-run-ovn\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:27.065723    5087 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:33:27.065725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065738    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-run-ovn\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065730    5087 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065720    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-modprobe-d\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:27.065759    5087 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-metrics-tls podName:2a216b8f-a750-4e1e-9ab1-ff6c89d416e5 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:27.565745673 +0000 UTC m=+2.040249663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-metrics-tls") pod "dns-default-5kfvz" (UID: "2a216b8f-a750-4e1e-9ab1-ff6c89d416e5") : secret "dns-default-metrics-tls" not found
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065791    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-run-ovn-kubernetes\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065799    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-modprobe-d\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065810    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065826    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-run-ovn-kubernetes\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065830    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-host\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065852    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/752a05d6-f630-42ec-9795-0d28e9ed7d80-cnibin\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065867    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-kubelet-config\") pod \"global-pull-secret-syncer-2z8sh\" (UID: \"2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49\") " pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065871    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065884    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-sysctl-d\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065907    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/752a05d6-f630-42ec-9795-0d28e9ed7d80-tuning-conf-dir\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065911    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/752a05d6-f630-42ec-9795-0d28e9ed7d80-cnibin\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065932    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-slash\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065945    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a724a1c0-c643-49ff-a555-5bddec4bfacb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065955    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-log-socket\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065963    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-slash\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065954    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-host\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065978    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-run\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065980    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/752a05d6-f630-42ec-9795-0d28e9ed7d80-tuning-conf-dir\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065983    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-log-socket\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066009    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-run\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066011    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-run-k8s-cni-cncf-io\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.065931    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49-kubelet-config\") pod \"global-pull-secret-syncer-2z8sh\" (UID: \"2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49\") " pod="kube-system/global-pull-secret-syncer-2z8sh"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066019    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-sysctl-d\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066034    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-hostroot\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066070    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-run-k8s-cni-cncf-io\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066079    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d5f25065-34c9-4724-8dff-9ace0890d565-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qx7zq\" (UID: \"d5f25065-34c9-4724-8dff-9ace0890d565\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qx7zq"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066103    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-hostroot\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066104    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-tls\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066138    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/239a77e8-635c-4388-ad79-1cb3b15129d8-hosts-file\") pod \"node-resolver-9xqln\" (UID: \"239a77e8-635c-4388-ad79-1cb3b15129d8\") " pod="openshift-dns/node-resolver-9xqln"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066165    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a724a1c0-c643-49ff-a555-5bddec4bfacb-crio-socket\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066187    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-run-netns\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066201    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/239a77e8-635c-4388-ad79-1cb3b15129d8-hosts-file\") pod \"node-resolver-9xqln\" (UID: \"239a77e8-635c-4388-ad79-1cb3b15129d8\") " pod="openshift-dns/node-resolver-9xqln"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066218    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a7ef1f3-4e9d-40db-93a3-4d38a3474e89-host-slash\") pod \"iptables-alerter-w9ckc\" (UID: \"2a7ef1f3-4e9d-40db-93a3-4d38a3474e89\") " pod="openshift-network-operator/iptables-alerter-w9ckc"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066246    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a724a1c0-c643-49ff-a555-5bddec4bfacb-crio-socket\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066246    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7986120b-482f-4b0b-a605-32770569962d-host-run-netns\") pod \"ovnkube-node-bjt99\" (UID: \"7986120b-482f-4b0b-a605-32770569962d\") " pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066247    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-lib-modules\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066284    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2a7ef1f3-4e9d-40db-93a3-4d38a3474e89-host-slash\") pod \"iptables-alerter-w9ckc\" (UID: \"2a7ef1f3-4e9d-40db-93a3-4d38a3474e89\") " pod="openshift-network-operator/iptables-alerter-w9ckc"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066306    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-run-multus-certs\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066335    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83917e8e-50b6-4c52-b337-6f1be5a2c26a-service-ca-bundle\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066339    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-run-multus-certs\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066386    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-systemd\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066412    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-cnibin\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066423    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-lib-modules\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066435    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-run-netns\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066457    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-systemd\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:27.066456    5087 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/83917e8e-50b6-4c52-b337-6f1be5a2c26a-service-ca-bundle podName:83917e8e-50b6-4c52-b337-6f1be5a2c26a nodeName:}" failed. No retries permitted until 2026-10-30 21:33:27.566438569 +0000 UTC m=+2.040942579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/83917e8e-50b6-4c52-b337-6f1be5a2c26a-service-ca-bundle") pod "router-default-54cbfc65d5-sg48h" (UID: "83917e8e-50b6-4c52-b337-6f1be5a2c26a") : configmap references non-existent config key: service-ca.crt
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066482    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-host-run-netns\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066497    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-sysconfig\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066501    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-cnibin\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066526    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-registration-dir\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066555    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-system-cni-dir\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066581    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/752a05d6-f630-42ec-9795-0d28e9ed7d80-system-cni-dir\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066588    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/061313cb-4d50-4ae5-afda-dc63a62a013b-etc-sysconfig\") pod \"tuned-86426\" (UID: \"061313cb-4d50-4ae5-afda-dc63a62a013b\") " pod="openshift-cluster-node-tuning-operator/tuned-86426"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066607    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-multus-cni-dir\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066610    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-registration-dir\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066644    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-multus-conf-dir\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066655    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-system-cni-dir\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066647    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/752a05d6-f630-42ec-9795-0d28e9ed7d80-system-cni-dir\") pod \"multus-additional-cni-plugins-956lw\" (UID: \"752a05d6-f630-42ec-9795-0d28e9ed7d80\") " pod="openshift-multus/multus-additional-cni-plugins-956lw"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066670    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066719    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066668    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-multus-cni-dir\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066679    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a1460f5-20f5-4bd1-843f-3838935f2ce2-multus-conf-dir\") pod \"multus-25lhq\" (UID: \"9a1460f5-20f5-4bd1-843f-3838935f2ce2\") " pod="openshift-multus/multus-25lhq"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066787    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-device-dir\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:33:27.067831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.066746    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/23a040b8-2420-4d59-b426-f52832dc7de1-device-dir\") pod \"aws-ebs-csi-driver-node-f9f4j\" (UID: \"23a040b8-2420-4d59-b426-f52832dc7de1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9f4j"
Oct 30 21:33:27.069907 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.068363    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-tls\") pod \"image-registry-74b44b76d6-l7p76\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") " pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:27.069907 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.068562    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/83917e8e-50b6-4c52-b337-6f1be5a2c26a-metrics-certs\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:27.069907 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.068616    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d5f25065-34c9-4724-8dff-9ace0890d565-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-qx7zq\" (UID: \"d5f25065-34c9-4724-8dff-9ace0890d565\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-qx7zq"
Oct 30 21:33:27.069907 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.069069    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-tls\") pod \"image-registry-7fc9466475-jz9vh\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") " pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:27.104527 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.104500    5087 scope.go:122] "RemoveContainer" containerID="d036f501bf39148b5ef1dfc318a42d74c69b78f8ea89630deb12c0797ede2521"
Oct 30 21:33:27.108762 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.108731    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:27.108959 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.108942    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-qx7zq"
Oct 30 21:33:27.114962 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.114941    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:27.548552 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.548519    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-qx7zq"]
Oct 30 21:33:27.551993 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.551964    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7fc9466475-jz9vh"]
Oct 30 21:33:27.552797 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:27.552763    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f25065_34c9_4724_8dff_9ace0890d565.slice/crio-ba6c7d934d688f260dd3f69adf711b7a6e3d818059ddafdcb1f1a6f70503d644 WatchSource:0}: Error finding container ba6c7d934d688f260dd3f69adf711b7a6e3d818059ddafdcb1f1a6f70503d644: Status 404 returned error can't find the container with id ba6c7d934d688f260dd3f69adf711b7a6e3d818059ddafdcb1f1a6f70503d644
Oct 30 21:33:27.555374 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.555163    5087 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:33:27.555479 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.555427    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74b44b76d6-l7p76"]
Oct 30 21:33:27.555786 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:27.555765    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42dc1fb7_b6e0_40d2_bf53_20079815b12d.slice/crio-993ef86e4edeac393959e3a25b8b27a6be355fbe898750ae4f421775c4c5a146 WatchSource:0}: Error finding container 993ef86e4edeac393959e3a25b8b27a6be355fbe898750ae4f421775c4c5a146: Status 404 returned error can't find the container with id 993ef86e4edeac393959e3a25b8b27a6be355fbe898750ae4f421775c4c5a146
Oct 30 21:33:27.561798 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.561768    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ljjfm_b7252dc2-b8ae-41f4-b7d8-c0ca9986f8e5/serve-healthcheck-canary/0.log"
Oct 30 21:33:27.571947 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.571925    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83917e8e-50b6-4c52-b337-6f1be5a2c26a-service-ca-bundle\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:27.572048 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.571977    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kgrcd\" (UID: \"f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd"
Oct 30 21:33:27.572048 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.572009    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a724a1c0-c643-49ff-a555-5bddec4bfacb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:27.572048 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.572038    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs\") pod \"network-metrics-daemon-bxbq6\" (UID: \"032b9335-657e-4a86-aa6d-4ef46163014a\") " pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:33:27.572209 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.572069    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2qz48\" (UID: \"9f16bd1f-a3bd-45ee-a02e-cb6b9e982327\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48"
Oct 30 21:33:27.572209 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:27.572093    5087 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/83917e8e-50b6-4c52-b337-6f1be5a2c26a-service-ca-bundle podName:83917e8e-50b6-4c52-b337-6f1be5a2c26a nodeName:}" failed. No retries permitted until 2026-10-30 21:33:28.572075283 +0000 UTC m=+3.046579247 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/83917e8e-50b6-4c52-b337-6f1be5a2c26a-service-ca-bundle") pod "router-default-54cbfc65d5-sg48h" (UID: "83917e8e-50b6-4c52-b337-6f1be5a2c26a") : configmap references non-existent config key: service-ca.crt
Oct 30 21:33:27.572209 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:27.572130    5087 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:33:27.572209 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:27.572142    5087 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:33:27.572209 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:27.572178    5087 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls podName:9f16bd1f-a3bd-45ee-a02e-cb6b9e982327 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:28.57216732 +0000 UTC m=+3.046671291 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2qz48" (UID: "9f16bd1f-a3bd-45ee-a02e-cb6b9e982327") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:33:27.572209 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:27.572182    5087 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:33:27.572209 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:27.572182    5087 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:33:27.572209 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:27.572201    5087 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-tls podName:f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf nodeName:}" failed. No retries permitted until 2026-10-30 21:33:28.572184705 +0000 UTC m=+3.046688669 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-kgrcd" (UID: "f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf") : secret "samples-operator-tls" not found
Oct 30 21:33:27.572209 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.572129    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-metrics-tls\") pod \"dns-default-5kfvz\" (UID: \"2a216b8f-a750-4e1e-9ab1-ff6c89d416e5\") " pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:27.572638 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:27.572219    5087 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-metrics-tls podName:2a216b8f-a750-4e1e-9ab1-ff6c89d416e5 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:28.572209789 +0000 UTC m=+3.046713751 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-metrics-tls") pod "dns-default-5kfvz" (UID: "2a216b8f-a750-4e1e-9ab1-ff6c89d416e5") : secret "dns-default-metrics-tls" not found
Oct 30 21:33:27.572638 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:27.572231    5087 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:33:27.572638 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:27.572249    5087 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a724a1c0-c643-49ff-a555-5bddec4bfacb-insights-runtime-extractor-tls podName:a724a1c0-c643-49ff-a555-5bddec4bfacb nodeName:}" failed. No retries permitted until 2026-10-30 21:33:28.572230989 +0000 UTC m=+3.046734958 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a724a1c0-c643-49ff-a555-5bddec4bfacb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g27zs" (UID: "a724a1c0-c643-49ff-a555-5bddec4bfacb") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:33:27.572638 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:27.572268    5087 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs podName:032b9335-657e-4a86-aa6d-4ef46163014a nodeName:}" failed. No retries permitted until 2026-10-30 21:33:28.572260523 +0000 UTC m=+3.046764486 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs") pod "network-metrics-daemon-bxbq6" (UID: "032b9335-657e-4a86-aa6d-4ef46163014a") : secret "metrics-daemon-secret" not found
Oct 30 21:33:27.602379 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:27.602338    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod392cc7ef_7cd2_4090_a246_0347d5af7bbf.slice/crio-8286f2dee2e87a47a7765219547a44effd40c7e5c6f940eebf1500e67a0cbdfe WatchSource:0}: Error finding container 8286f2dee2e87a47a7765219547a44effd40c7e5c6f940eebf1500e67a0cbdfe: Status 404 returned error can't find the container with id 8286f2dee2e87a47a7765219547a44effd40c7e5c6f940eebf1500e67a0cbdfe
Oct 30 21:33:27.761668 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.761638    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-fjrtf_3d0811a7-4a83-4758-8ca0-2ac7dd5a9408/migrator/0.log"
Oct 30 21:33:27.781020 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.780978    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fc9466475-jz9vh" event={"ID":"42dc1fb7-b6e0-40d2-bf53-20079815b12d","Type":"ContainerStarted","Data":"df6b3632666c81970dcfd5816025ba4d7e899974efd8e6fd819e90683624a69e"}
Oct 30 21:33:27.781192 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.781030    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fc9466475-jz9vh" event={"ID":"42dc1fb7-b6e0-40d2-bf53-20079815b12d","Type":"ContainerStarted","Data":"993ef86e4edeac393959e3a25b8b27a6be355fbe898750ae4f421775c4c5a146"}
Oct 30 21:33:27.783129 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.783109    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 21:33:27.783255 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.783189    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf" event={"ID":"1196dab1-80d7-48eb-8d05-1a26ab6d0759","Type":"ContainerStarted","Data":"dd5f0edd7d03b4c4c3f761b9e63abbee9cf549e7bd683c7e9f7c048c0bd95c55"}
Oct 30 21:33:27.784767 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.784735    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-qx7zq" event={"ID":"d5f25065-34c9-4724-8dff-9ace0890d565","Type":"ContainerStarted","Data":"ba6c7d934d688f260dd3f69adf711b7a6e3d818059ddafdcb1f1a6f70503d644"}
Oct 30 21:33:27.786565 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.786543    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74b44b76d6-l7p76" event={"ID":"392cc7ef-7cd2-4090-a246-0347d5af7bbf","Type":"ContainerStarted","Data":"3cad43653aef065ab53061fc1fa622173077ffa91fa66e07fc1ebd44ac2a72c9"}
Oct 30 21:33:27.786668 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.786585    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74b44b76d6-l7p76" event={"ID":"392cc7ef-7cd2-4090-a246-0347d5af7bbf","Type":"ContainerStarted","Data":"8286f2dee2e87a47a7765219547a44effd40c7e5c6f940eebf1500e67a0cbdfe"}
Oct 30 21:33:27.961948 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:27.961866    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-mf2wv_298f2a29-a322-4977-bd9f-d6b765eff5d1/migrator/0.log"
Oct 30 21:33:28.164932 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:28.164901    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc_25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245/kube-storage-version-migrator-operator/0.log"
Oct 30 21:33:28.483526 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:28.483496    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h5xkl"
Oct 30 21:33:28.580524 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:28.580495    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kgrcd\" (UID: \"f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd"
Oct 30 21:33:28.580686 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:28.580532    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a724a1c0-c643-49ff-a555-5bddec4bfacb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:28.580686 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:28.580562    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs\") pod \"network-metrics-daemon-bxbq6\" (UID: \"032b9335-657e-4a86-aa6d-4ef46163014a\") " pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:33:28.580686 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:28.580591    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2qz48\" (UID: \"9f16bd1f-a3bd-45ee-a02e-cb6b9e982327\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48"
Oct 30 21:33:28.580686 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:28.580621    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-metrics-tls\") pod \"dns-default-5kfvz\" (UID: \"2a216b8f-a750-4e1e-9ab1-ff6c89d416e5\") " pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:28.580686 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:28.580652    5087 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 30 21:33:28.580686 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:28.580668    5087 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:33:28.580686 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:28.580677    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83917e8e-50b6-4c52-b337-6f1be5a2c26a-service-ca-bundle\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:28.581021 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:28.580725    5087 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-tls podName:f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf nodeName:}" failed. No retries permitted until 2026-10-30 21:33:30.580705667 +0000 UTC m=+5.055209631 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-kgrcd" (UID: "f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf") : secret "samples-operator-tls" not found
Oct 30 21:33:28.581021 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:28.580795    5087 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/83917e8e-50b6-4c52-b337-6f1be5a2c26a-service-ca-bundle podName:83917e8e-50b6-4c52-b337-6f1be5a2c26a nodeName:}" failed. No retries permitted until 2026-10-30 21:33:30.580779719 +0000 UTC m=+5.055283696 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/83917e8e-50b6-4c52-b337-6f1be5a2c26a-service-ca-bundle") pod "router-default-54cbfc65d5-sg48h" (UID: "83917e8e-50b6-4c52-b337-6f1be5a2c26a") : configmap references non-existent config key: service-ca.crt
Oct 30 21:33:28.581021 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:28.580817    5087 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a724a1c0-c643-49ff-a555-5bddec4bfacb-insights-runtime-extractor-tls podName:a724a1c0-c643-49ff-a555-5bddec4bfacb nodeName:}" failed. No retries permitted until 2026-10-30 21:33:30.580807113 +0000 UTC m=+5.055311075 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a724a1c0-c643-49ff-a555-5bddec4bfacb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g27zs" (UID: "a724a1c0-c643-49ff-a555-5bddec4bfacb") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:33:28.581021 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:28.580871    5087 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:33:28.581021 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:28.580904    5087 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs podName:032b9335-657e-4a86-aa6d-4ef46163014a nodeName:}" failed. No retries permitted until 2026-10-30 21:33:30.580893543 +0000 UTC m=+5.055397530 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs") pod "network-metrics-daemon-bxbq6" (UID: "032b9335-657e-4a86-aa6d-4ef46163014a") : secret "metrics-daemon-secret" not found
Oct 30 21:33:28.581021 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:28.580944    5087 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:33:28.581021 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:28.580973    5087 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls podName:9f16bd1f-a3bd-45ee-a02e-cb6b9e982327 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:30.580963661 +0000 UTC m=+5.055467652 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2qz48" (UID: "9f16bd1f-a3bd-45ee-a02e-cb6b9e982327") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:33:28.583705 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:28.583659    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2a216b8f-a750-4e1e-9ab1-ff6c89d416e5-metrics-tls\") pod \"dns-default-5kfvz\" (UID: \"2a216b8f-a750-4e1e-9ab1-ff6c89d416e5\") " pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:28.615968 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:28.615941    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:28.790108 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:28.790028    5087 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:33:28.790547 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:28.790034    5087 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:33:28.998688 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:28.998655    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5kfvz"]
Oct 30 21:33:29.001937 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:29.001902    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a216b8f_a750_4e1e_9ab1_ff6c89d416e5.slice/crio-3c6c50258020de49c96049bf3ddead7e07fefd437e0a4cb4847ae35406225a23 WatchSource:0}: Error finding container 3c6c50258020de49c96049bf3ddead7e07fefd437e0a4cb4847ae35406225a23: Status 404 returned error can't find the container with id 3c6c50258020de49c96049bf3ddead7e07fefd437e0a4cb4847ae35406225a23
Oct 30 21:33:29.795933 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:29.795893    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-qx7zq" event={"ID":"d5f25065-34c9-4724-8dff-9ace0890d565","Type":"ContainerStarted","Data":"da0787e072ee2cfe1b564f2bdb4a4346f1c42e069bc4c9efd58a45c1dd6ec57e"}
Oct 30 21:33:29.797163 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:29.797135    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5kfvz" event={"ID":"2a216b8f-a750-4e1e-9ab1-ff6c89d416e5","Type":"ContainerStarted","Data":"3c6c50258020de49c96049bf3ddead7e07fefd437e0a4cb4847ae35406225a23"}
Oct 30 21:33:30.597829 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:30.597790    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83917e8e-50b6-4c52-b337-6f1be5a2c26a-service-ca-bundle\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:30.598036 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:30.597883    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kgrcd\" (UID: \"f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd"
Oct 30 21:33:30.598036 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:30.597913    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a724a1c0-c643-49ff-a555-5bddec4bfacb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:30.598036 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:30.597939    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs\") pod \"network-metrics-daemon-bxbq6\" (UID: \"032b9335-657e-4a86-aa6d-4ef46163014a\") " pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:33:30.598036 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:30.597972    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2qz48\" (UID: \"9f16bd1f-a3bd-45ee-a02e-cb6b9e982327\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48"
Oct 30 21:33:30.598271 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:30.598037    5087 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:33:30.598271 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:30.598110    5087 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a724a1c0-c643-49ff-a555-5bddec4bfacb-insights-runtime-extractor-tls podName:a724a1c0-c643-49ff-a555-5bddec4bfacb nodeName:}" failed. No retries permitted until 2026-10-30 21:33:34.598093407 +0000 UTC m=+9.072597364 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a724a1c0-c643-49ff-a555-5bddec4bfacb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g27zs" (UID: "a724a1c0-c643-49ff-a555-5bddec4bfacb") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:33:30.598486 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:30.598461    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83917e8e-50b6-4c52-b337-6f1be5a2c26a-service-ca-bundle\") pod \"router-default-54cbfc65d5-sg48h\" (UID: \"83917e8e-50b6-4c52-b337-6f1be5a2c26a\") " pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:30.598559 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:30.598498    5087 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 21:33:30.598604 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:30.598558    5087 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs podName:032b9335-657e-4a86-aa6d-4ef46163014a nodeName:}" failed. No retries permitted until 2026-10-30 21:33:34.598541762 +0000 UTC m=+9.073045720 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs") pod "network-metrics-daemon-bxbq6" (UID: "032b9335-657e-4a86-aa6d-4ef46163014a") : secret "metrics-daemon-secret" not found
Oct 30 21:33:30.598604 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:30.598578    5087 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 30 21:33:30.598703 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:30.598650    5087 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls podName:9f16bd1f-a3bd-45ee-a02e-cb6b9e982327 nodeName:}" failed. No retries permitted until 2026-10-30 21:33:34.598633115 +0000 UTC m=+9.073137086 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2qz48" (UID: "9f16bd1f-a3bd-45ee-a02e-cb6b9e982327") : secret "cluster-monitoring-operator-tls" not found
Oct 30 21:33:30.600885 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:30.600859    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-kgrcd\" (UID: \"f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd"
Oct 30 21:33:30.703679 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:30.703634    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:30.703874 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:30.703712    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd"
Oct 30 21:33:31.184114 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:31.184086    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd"]
Oct 30 21:33:31.187489 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:31.187467    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-54cbfc65d5-sg48h"]
Oct 30 21:33:31.236503 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:31.236474    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83917e8e_50b6_4c52_b337_6f1be5a2c26a.slice/crio-6df19b365d95ca333e0233ad535aa742b3c3715f9b479996b4c0ff91eb55d6c4 WatchSource:0}: Error finding container 6df19b365d95ca333e0233ad535aa742b3c3715f9b479996b4c0ff91eb55d6c4: Status 404 returned error can't find the container with id 6df19b365d95ca333e0233ad535aa742b3c3715f9b479996b4c0ff91eb55d6c4
Oct 30 21:33:31.419396 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:31.419309    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:31.808870 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:31.808831    5087 generic.go:356] "Generic (PLEG): container finished" podID="f54f9754-2024-41b9-8f84-fc3d02a22253" containerID="4306113a218453fefd6fd63545507425511554f8c6b7bc99a74906a97d37cdb7" exitCode=0
Oct 30 21:33:31.809125 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:31.808903    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-z4jbc" event={"ID":"f54f9754-2024-41b9-8f84-fc3d02a22253","Type":"ContainerDied","Data":"4306113a218453fefd6fd63545507425511554f8c6b7bc99a74906a97d37cdb7"}
Oct 30 21:33:31.809321 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:31.809300    5087 scope.go:122] "RemoveContainer" containerID="4306113a218453fefd6fd63545507425511554f8c6b7bc99a74906a97d37cdb7"
Oct 30 21:33:31.811035 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:31.811007    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5kfvz" event={"ID":"2a216b8f-a750-4e1e-9ab1-ff6c89d416e5","Type":"ContainerStarted","Data":"ab3fa789421bb80fe37d3947bc4f080fe70fd3be43777a0cdbfd2c9b2775f630"}
Oct 30 21:33:31.811116 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:31.811058    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5kfvz" event={"ID":"2a216b8f-a750-4e1e-9ab1-ff6c89d416e5","Type":"ContainerStarted","Data":"2dce27ce96d6c103775f972a4994f07b8f597acd55f73da492596a550dada594"}
Oct 30 21:33:31.811116 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:31.811087    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:31.812301 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:31.812277    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd" event={"ID":"f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf","Type":"ContainerStarted","Data":"b778bb26f303ade56f45c7dd5c26653fe2752a5eb0fc9207e7674d43b289abda"}
Oct 30 21:33:31.813868 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:31.813836    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-54cbfc65d5-sg48h" event={"ID":"83917e8e-50b6-4c52-b337-6f1be5a2c26a","Type":"ContainerStarted","Data":"74ff30bffde853335de8eb5291ce1ec7653153ed60290cc6ae4bff96e659bccc"}
Oct 30 21:33:31.813954 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:31.813886    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-54cbfc65d5-sg48h" event={"ID":"83917e8e-50b6-4c52-b337-6f1be5a2c26a","Type":"ContainerStarted","Data":"6df19b365d95ca333e0233ad535aa742b3c3715f9b479996b4c0ff91eb55d6c4"}
Oct 30 21:33:32.704104 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:32.704060    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:32.708686 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:32.708664    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:32.821339 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:32.821297    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-z4jbc" event={"ID":"f54f9754-2024-41b9-8f84-fc3d02a22253","Type":"ContainerStarted","Data":"68fda935f50390828e5b0b15cd1fecf09044d64fa21b6f90b98578a9212c48eb"}
Oct 30 21:33:32.833430 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:32.833394    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf"
Oct 30 21:33:32.833603 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:32.833502    5087 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:33:32.837684 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:32.837659    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-z8clf"
Oct 30 21:33:33.829705 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:33.829667    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd" event={"ID":"f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf","Type":"ContainerStarted","Data":"3506d68111400de16048558c3422e4f530ee74e63a8b7c80878582bf562f18f2"}
Oct 30 21:33:33.830081 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:33.829732    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-kgrcd" event={"ID":"f9ba7aaf-d5bc-4fce-9ecd-0de3d9fcacbf","Type":"ContainerStarted","Data":"56f17ab8522264e05a1e3b9926a74316b9fcb62c3ecec9fccfdb62b75b46bded"}
Oct 30 21:33:33.830081 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:33.829854    5087 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:33:33.877253 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:33.877224    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:34.614856 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:34.614830    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:34.615072 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:34.615058    5087 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:33:34.627039 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:34.627014    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2qz48\" (UID: \"9f16bd1f-a3bd-45ee-a02e-cb6b9e982327\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48"
Oct 30 21:33:34.627194 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:34.627170    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a724a1c0-c643-49ff-a555-5bddec4bfacb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:34.627260 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:34.627212    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs\") pod \"network-metrics-daemon-bxbq6\" (UID: \"032b9335-657e-4a86-aa6d-4ef46163014a\") " pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:33:34.627481 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:34.627460    5087 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:33:34.627558 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:34.627528    5087 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a724a1c0-c643-49ff-a555-5bddec4bfacb-insights-runtime-extractor-tls podName:a724a1c0-c643-49ff-a555-5bddec4bfacb nodeName:}" failed. No retries permitted until 2026-10-30 21:33:42.627507971 +0000 UTC m=+17.102011949 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a724a1c0-c643-49ff-a555-5bddec4bfacb-insights-runtime-extractor-tls") pod "insights-runtime-extractor-g27zs" (UID: "a724a1c0-c643-49ff-a555-5bddec4bfacb") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:33:34.629657 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:34.629631    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/032b9335-657e-4a86-aa6d-4ef46163014a-metrics-certs\") pod \"network-metrics-daemon-bxbq6\" (UID: \"032b9335-657e-4a86-aa6d-4ef46163014a\") " pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:33:34.629791 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:34.629667    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f16bd1f-a3bd-45ee-a02e-cb6b9e982327-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2qz48\" (UID: \"9f16bd1f-a3bd-45ee-a02e-cb6b9e982327\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48"
Oct 30 21:33:34.902131 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:34.902061    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48"
Oct 30 21:33:34.914143 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:34.914114    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxbq6"
Oct 30 21:33:35.114148 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:35.114106    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:35.285554 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:35.285525    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48"]
Oct 30 21:33:35.286567 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:35.286546    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:33:35.288309 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:35.288283    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f16bd1f_a3bd_45ee_a02e_cb6b9e982327.slice/crio-2ce07fbbcf4cf95a6aebff24b9a70fc7325c14f5bc48f978ddb335446b714c53 WatchSource:0}: Error finding container 2ce07fbbcf4cf95a6aebff24b9a70fc7325c14f5bc48f978ddb335446b714c53: Status 404 returned error can't find the container with id 2ce07fbbcf4cf95a6aebff24b9a70fc7325c14f5bc48f978ddb335446b714c53
Oct 30 21:33:35.334581 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:35.334550    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bxbq6"]
Oct 30 21:33:35.337753 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:35.337723    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod032b9335_657e_4a86_aa6d_4ef46163014a.slice/crio-7a6922fea0eb1b23b5e0779c7ea3dbcf0c7694a2b953ad742012aacb37d2c4f3 WatchSource:0}: Error finding container 7a6922fea0eb1b23b5e0779c7ea3dbcf0c7694a2b953ad742012aacb37d2c4f3: Status 404 returned error can't find the container with id 7a6922fea0eb1b23b5e0779c7ea3dbcf0c7694a2b953ad742012aacb37d2c4f3
Oct 30 21:33:35.837877 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:35.837836    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48" event={"ID":"9f16bd1f-a3bd-45ee-a02e-cb6b9e982327","Type":"ContainerStarted","Data":"2ce07fbbcf4cf95a6aebff24b9a70fc7325c14f5bc48f978ddb335446b714c53"}
Oct 30 21:33:35.839149 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:35.839118    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bxbq6" event={"ID":"032b9335-657e-4a86-aa6d-4ef46163014a","Type":"ContainerStarted","Data":"7a6922fea0eb1b23b5e0779c7ea3dbcf0c7694a2b953ad742012aacb37d2c4f3"}
Oct 30 21:33:37.404434 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:37.404399    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bjt99"
Oct 30 21:33:37.560291 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:37.560262    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:37.560489 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:37.560425    5087 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:33:37.561604 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:37.561585    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-54cbfc65d5-sg48h"
Oct 30 21:33:37.849480 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:37.849442    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2qz48" event={"ID":"9f16bd1f-a3bd-45ee-a02e-cb6b9e982327","Type":"ContainerStarted","Data":"9241d91b13600426733ccc16b607f2164fa5e4a1cac183535d1f5b4d39defd51"}
Oct 30 21:33:37.851364 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:37.851315    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bxbq6" event={"ID":"032b9335-657e-4a86-aa6d-4ef46163014a","Type":"ContainerStarted","Data":"939c6ba0070e3833c2b78b8e35f96b2137c57881b559716032d64b8120881961"}
Oct 30 21:33:37.851516 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:37.851380    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bxbq6" event={"ID":"032b9335-657e-4a86-aa6d-4ef46163014a","Type":"ContainerStarted","Data":"f4c34c0f499657f790befb0e3d7a0574dd1bd249bac8b2f2b5744b4dcc5d2a08"}
Oct 30 21:33:41.210648 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:41.210609    5087 patch_prober.go:29] interesting pod/image-registry-7fc9466475-jz9vh container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 21:33:41.211020 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:41.210667    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7fc9466475-jz9vh" podUID="42dc1fb7-b6e0-40d2-bf53-20079815b12d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:33:41.824026 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:41.823992    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5kfvz"
Oct 30 21:33:42.696614 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:42.696579    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a724a1c0-c643-49ff-a555-5bddec4bfacb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:42.698945 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:42.698927    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a724a1c0-c643-49ff-a555-5bddec4bfacb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-g27zs\" (UID: \"a724a1c0-c643-49ff-a555-5bddec4bfacb\") " pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:42.718195 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:42.718160    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-g27zs"
Oct 30 21:33:43.086074 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:43.084222    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-g27zs"]
Oct 30 21:33:43.119991 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:43.119957    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda724a1c0_c643_49ff_a555_5bddec4bfacb.slice/crio-c37b7fdb90e6f9499356ad52b0aabc7411b55d5d631e28ff6bdd93900daa0f24 WatchSource:0}: Error finding container c37b7fdb90e6f9499356ad52b0aabc7411b55d5d631e28ff6bdd93900daa0f24: Status 404 returned error can't find the container with id c37b7fdb90e6f9499356ad52b0aabc7411b55d5d631e28ff6bdd93900daa0f24
Oct 30 21:33:43.876335 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:43.876298    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g27zs" event={"ID":"a724a1c0-c643-49ff-a555-5bddec4bfacb","Type":"ContainerStarted","Data":"206ae1022fa33111833caade328572cff779f536f17e4d885a1d4d00be4ac0ca"}
Oct 30 21:33:43.876714 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:43.876376    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g27zs" event={"ID":"a724a1c0-c643-49ff-a555-5bddec4bfacb","Type":"ContainerStarted","Data":"c37b7fdb90e6f9499356ad52b0aabc7411b55d5d631e28ff6bdd93900daa0f24"}
Oct 30 21:33:43.882138 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:43.882115    5087 patch_prober.go:29] interesting pod/image-registry-74b44b76d6-l7p76 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 21:33:43.882254 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:43.882163    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-74b44b76d6-l7p76" podUID="392cc7ef-7cd2-4090-a246-0347d5af7bbf" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:33:44.882552 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:44.882506    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g27zs" event={"ID":"a724a1c0-c643-49ff-a555-5bddec4bfacb","Type":"ContainerStarted","Data":"ea1c0663476a8893b6b792e90a62b838041c35979c0f91e48c97cbf6b6cedbc1"}
Oct 30 21:33:45.119176 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.119145    5087 patch_prober.go:29] interesting pod/image-registry-7fc9466475-jz9vh container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 21:33:45.119323 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.119196    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7fc9466475-jz9vh" podUID="42dc1fb7-b6e0-40d2-bf53-20079815b12d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:33:45.176745 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.176719    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hmplx"]
Oct 30 21:33:45.178990 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.178975    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hmplx"
Oct 30 21:33:45.214463 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.214433    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7589a4df-2be1-47b1-836f-09ba72813d82-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-hmplx\" (UID: \"7589a4df-2be1-47b1-836f-09ba72813d82\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hmplx"
Oct 30 21:33:45.226168 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.226145    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-2lvn8"]
Oct 30 21:33:45.228467 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.228449    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-2lvn8"
Oct 30 21:33:45.233959 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.233935    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7fc9466475-jz9vh"]
Oct 30 21:33:45.239850 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.239825    5087 patch_prober.go:29] interesting pod/image-registry-7fc9466475-jz9vh container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 21:33:45.239955 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.239895    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7fc9466475-jz9vh" podUID="42dc1fb7-b6e0-40d2-bf53-20079815b12d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:33:45.274728 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.274698    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 21:33:45.274728 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.274698    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-2v42s\""
Oct 30 21:33:45.281626 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.281602    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-588c778845-rvqdr"]
Oct 30 21:33:45.283667 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.283653    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-588c778845-rvqdr"
Oct 30 21:33:45.284004 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.283990    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 21:33:45.284143 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.284130    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 21:33:45.315797 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.315774    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8717a682-defa-4983-8106-b075a1d20dd3-image-registry-private-configuration\") pod \"image-registry-588c778845-rvqdr\" (UID: \"8717a682-defa-4983-8106-b075a1d20dd3\") " pod="openshift-image-registry/image-registry-588c778845-rvqdr"
Oct 30 21:33:45.315898 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.315803    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8717a682-defa-4983-8106-b075a1d20dd3-registry-tls\") pod \"image-registry-588c778845-rvqdr\" (UID: \"8717a682-defa-4983-8106-b075a1d20dd3\") " pod="openshift-image-registry/image-registry-588c778845-rvqdr"
Oct 30 21:33:45.315898 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.315824    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c873b45-e105-43e0-a528-9b327f68114c-tmp\") pod \"downloads-6dd789c8b7-2lvn8\" (UID: \"1c873b45-e105-43e0-a528-9b327f68114c\") " pod="openshift-console/downloads-6dd789c8b7-2lvn8"
Oct 30 21:33:45.315898 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.315882    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8717a682-defa-4983-8106-b075a1d20dd3-bound-sa-token\") pod \"image-registry-588c778845-rvqdr\" (UID: \"8717a682-defa-4983-8106-b075a1d20dd3\") " pod="openshift-image-registry/image-registry-588c778845-rvqdr"
Oct 30 21:33:45.316021 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.316007    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8717a682-defa-4983-8106-b075a1d20dd3-installation-pull-secrets\") pod \"image-registry-588c778845-rvqdr\" (UID: \"8717a682-defa-4983-8106-b075a1d20dd3\") " pod="openshift-image-registry/image-registry-588c778845-rvqdr"
Oct 30 21:33:45.316104 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.316041    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8717a682-defa-4983-8106-b075a1d20dd3-ca-trust-extracted\") pod \"image-registry-588c778845-rvqdr\" (UID: \"8717a682-defa-4983-8106-b075a1d20dd3\") " pod="openshift-image-registry/image-registry-588c778845-rvqdr"
Oct 30 21:33:45.316104 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.316081    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8717a682-defa-4983-8106-b075a1d20dd3-trusted-ca\") pod \"image-registry-588c778845-rvqdr\" (UID: \"8717a682-defa-4983-8106-b075a1d20dd3\") " pod="openshift-image-registry/image-registry-588c778845-rvqdr"
Oct 30 21:33:45.316104 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.316103    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7589a4df-2be1-47b1-836f-09ba72813d82-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-hmplx\" (UID: \"7589a4df-2be1-47b1-836f-09ba72813d82\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hmplx"
Oct 30 21:33:45.316220 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.316203    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvxmv\" (UniqueName: \"kubernetes.io/projected/1c873b45-e105-43e0-a528-9b327f68114c-kube-api-access-hvxmv\") pod \"downloads-6dd789c8b7-2lvn8\" (UID: \"1c873b45-e105-43e0-a528-9b327f68114c\") " pod="openshift-console/downloads-6dd789c8b7-2lvn8"
Oct 30 21:33:45.316265 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.316242    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8717a682-defa-4983-8106-b075a1d20dd3-registry-certificates\") pod \"image-registry-588c778845-rvqdr\" (UID: \"8717a682-defa-4983-8106-b075a1d20dd3\") " pod="openshift-image-registry/image-registry-588c778845-rvqdr"
Oct 30 21:33:45.316318 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.316267    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6mhw\" (UniqueName: \"kubernetes.io/projected/8717a682-defa-4983-8106-b075a1d20dd3-kube-api-access-t6mhw\") pod \"image-registry-588c778845-rvqdr\" (UID: \"8717a682-defa-4983-8106-b075a1d20dd3\") " pod="openshift-image-registry/image-registry-588c778845-rvqdr"
Oct 30 21:33:45.318564 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.318543    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/7589a4df-2be1-47b1-836f-09ba72813d82-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-hmplx\" (UID: \"7589a4df-2be1-47b1-836f-09ba72813d82\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hmplx"
Oct 30 21:33:45.329645 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.329623    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hmplx"]
Oct 30 21:33:45.330672 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.330651    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-2lvn8"]
Oct 30 21:33:45.338319 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.338300    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-bj628\""
Oct 30 21:33:45.375896 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.375859    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-588c778845-rvqdr"]
Oct 30 21:33:45.416767 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.416698    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvxmv\" (UniqueName: \"kubernetes.io/projected/1c873b45-e105-43e0-a528-9b327f68114c-kube-api-access-hvxmv\") pod \"downloads-6dd789c8b7-2lvn8\" (UID: \"1c873b45-e105-43e0-a528-9b327f68114c\") " pod="openshift-console/downloads-6dd789c8b7-2lvn8"
Oct 30 21:33:45.416767 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.416729    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8717a682-defa-4983-8106-b075a1d20dd3-registry-certificates\") pod \"image-registry-588c778845-rvqdr\" (UID: \"8717a682-defa-4983-8106-b075a1d20dd3\") " pod="openshift-image-registry/image-registry-588c778845-rvqdr"
Oct 30 21:33:45.416767 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.416748    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6mhw\" (UniqueName: \"kubernetes.io/projected/8717a682-defa-4983-8106-b075a1d20dd3-kube-api-access-t6mhw\") pod \"image-registry-588c778845-rvqdr\" (UID: \"8717a682-defa-4983-8106-b075a1d20dd3\") " pod="openshift-image-registry/image-registry-588c778845-rvqdr"
Oct 30 21:33:45.417020 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.416783    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8717a682-defa-4983-8106-b075a1d20dd3-image-registry-private-configuration\") pod \"image-registry-588c778845-rvqdr\" (UID: \"8717a682-defa-4983-8106-b075a1d20dd3\") " pod="openshift-image-registry/image-registry-588c778845-rvqdr"
Oct 30 21:33:45.417020 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.416800    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8717a682-defa-4983-8106-b075a1d20dd3-registry-tls\") pod \"image-registry-588c778845-rvqdr\" (UID: \"8717a682-defa-4983-8106-b075a1d20dd3\") " pod="openshift-image-registry/image-registry-588c778845-rvqdr"
Oct 30 21:33:45.417020 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.416822    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c873b45-e105-43e0-a528-9b327f68114c-tmp\") pod \"downloads-6dd789c8b7-2lvn8\" (UID: \"1c873b45-e105-43e0-a528-9b327f68114c\") " pod="openshift-console/downloads-6dd789c8b7-2lvn8"
Oct 30 21:33:45.417020 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.416842    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8717a682-defa-4983-8106-b075a1d20dd3-bound-sa-token\") pod \"image-registry-588c778845-rvqdr\" (UID: \"8717a682-defa-4983-8106-b075a1d20dd3\") " pod="openshift-image-registry/image-registry-588c778845-rvqdr"
Oct 30 21:33:45.417020 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.416886    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8717a682-defa-4983-8106-b075a1d20dd3-installation-pull-secrets\") pod \"image-registry-588c778845-rvqdr\" (UID: \"8717a682-defa-4983-8106-b075a1d20dd3\") " pod="openshift-image-registry/image-registry-588c778845-rvqdr"
Oct 30 21:33:45.417020 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.416926    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8717a682-defa-4983-8106-b075a1d20dd3-ca-trust-extracted\") pod \"image-registry-588c778845-rvqdr\" (UID: \"8717a682-defa-4983-8106-b075a1d20dd3\") " pod="openshift-image-registry/image-registry-588c778845-rvqdr"
Oct 30 21:33:45.417020 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.416953    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8717a682-defa-4983-8106-b075a1d20dd3-trusted-ca\") pod \"image-registry-588c778845-rvqdr\" (UID: \"8717a682-defa-4983-8106-b075a1d20dd3\") " pod="openshift-image-registry/image-registry-588c778845-rvqdr"
Oct 30 21:33:45.417402 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.417264    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1c873b45-e105-43e0-a528-9b327f68114c-tmp\") pod \"downloads-6dd789c8b7-2lvn8\" (UID: \"1c873b45-e105-43e0-a528-9b327f68114c\") " pod="openshift-console/downloads-6dd789c8b7-2lvn8"
Oct 30 21:33:45.417533 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.417507    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8717a682-defa-4983-8106-b075a1d20dd3-ca-trust-extracted\") pod \"image-registry-588c778845-rvqdr\" (UID: \"8717a682-defa-4983-8106-b075a1d20dd3\") " pod="openshift-image-registry/image-registry-588c778845-rvqdr"
Oct 30 21:33:45.417688 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.417667    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8717a682-defa-4983-8106-b075a1d20dd3-registry-certificates\") pod \"image-registry-588c778845-rvqdr\" (UID: \"8717a682-defa-4983-8106-b075a1d20dd3\") " pod="openshift-image-registry/image-registry-588c778845-rvqdr"
Oct 30 21:33:45.417808 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.417789    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8717a682-defa-4983-8106-b075a1d20dd3-trusted-ca\") pod \"image-registry-588c778845-rvqdr\" (UID: \"8717a682-defa-4983-8106-b075a1d20dd3\") " pod="openshift-image-registry/image-registry-588c778845-rvqdr"
Oct 30 21:33:45.419606 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.419583    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8717a682-defa-4983-8106-b075a1d20dd3-installation-pull-secrets\") pod \"image-registry-588c778845-rvqdr\" (UID: \"8717a682-defa-4983-8106-b075a1d20dd3\") " pod="openshift-image-registry/image-registry-588c778845-rvqdr"
Oct 30 21:33:45.419953 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.419934    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8717a682-defa-4983-8106-b075a1d20dd3-image-registry-private-configuration\") pod \"image-registry-588c778845-rvqdr\" (UID: \"8717a682-defa-4983-8106-b075a1d20dd3\") " pod="openshift-image-registry/image-registry-588c778845-rvqdr"
Oct 30 21:33:45.420141 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.420122    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8717a682-defa-4983-8106-b075a1d20dd3-registry-tls\") pod \"image-registry-588c778845-rvqdr\" (UID: \"8717a682-defa-4983-8106-b075a1d20dd3\") " pod="openshift-image-registry/image-registry-588c778845-rvqdr"
Oct 30 21:33:45.476286 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.476254    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6mhw\" (UniqueName: \"kubernetes.io/projected/8717a682-defa-4983-8106-b075a1d20dd3-kube-api-access-t6mhw\") pod \"image-registry-588c778845-rvqdr\" (UID: \"8717a682-defa-4983-8106-b075a1d20dd3\") " pod="openshift-image-registry/image-registry-588c778845-rvqdr"
Oct 30 21:33:45.476286 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.476279    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvxmv\" (UniqueName: \"kubernetes.io/projected/1c873b45-e105-43e0-a528-9b327f68114c-kube-api-access-hvxmv\") pod \"downloads-6dd789c8b7-2lvn8\" (UID: \"1c873b45-e105-43e0-a528-9b327f68114c\") " pod="openshift-console/downloads-6dd789c8b7-2lvn8"
Oct 30 21:33:45.477509 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.477487    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8717a682-defa-4983-8106-b075a1d20dd3-bound-sa-token\") pod \"image-registry-588c778845-rvqdr\" (UID: \"8717a682-defa-4983-8106-b075a1d20dd3\") " pod="openshift-image-registry/image-registry-588c778845-rvqdr"
Oct 30 21:33:45.491729 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.491704    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hmplx"
Oct 30 21:33:45.541484 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.541400    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-2lvn8"
Oct 30 21:33:45.594407 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.594180    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-588c778845-rvqdr"
Oct 30 21:33:45.691480 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.691406    5087 patch_prober.go:29] interesting pod/image-registry-74b44b76d6-l7p76 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 21:33:45.691620 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.691472    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-74b44b76d6-l7p76" podUID="392cc7ef-7cd2-4090-a246-0347d5af7bbf" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:33:45.874537 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.874499    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hmplx"]
Oct 30 21:33:45.878804 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:45.878772    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7589a4df_2be1_47b1_836f_09ba72813d82.slice/crio-c8410cf89953af12fb3042f20b91e6bec7515a59134bc19d49eaee4ffc8f5bc5 WatchSource:0}: Error finding container c8410cf89953af12fb3042f20b91e6bec7515a59134bc19d49eaee4ffc8f5bc5: Status 404 returned error can't find the container with id c8410cf89953af12fb3042f20b91e6bec7515a59134bc19d49eaee4ffc8f5bc5
Oct 30 21:33:45.888646 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.888613    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hmplx" event={"ID":"7589a4df-2be1-47b1-836f-09ba72813d82","Type":"ContainerStarted","Data":"c8410cf89953af12fb3042f20b91e6bec7515a59134bc19d49eaee4ffc8f5bc5"}
Oct 30 21:33:45.936768 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.936730    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-2lvn8"]
Oct 30 21:33:45.939890 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:45.939866    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c873b45_e105_43e0_a528_9b327f68114c.slice/crio-8902990444cd3410b7c1c0f423fedfa095061c163dcdad217f4c090a6c1aedde WatchSource:0}: Error finding container 8902990444cd3410b7c1c0f423fedfa095061c163dcdad217f4c090a6c1aedde: Status 404 returned error can't find the container with id 8902990444cd3410b7c1c0f423fedfa095061c163dcdad217f4c090a6c1aedde
Oct 30 21:33:45.988682 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:45.988652    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-588c778845-rvqdr"]
Oct 30 21:33:45.992707 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:45.992671    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8717a682_defa_4983_8106_b075a1d20dd3.slice/crio-760b012ffc49a0080251a610aedae606a80ee5124929d76937d0572a0b3a4a11 WatchSource:0}: Error finding container 760b012ffc49a0080251a610aedae606a80ee5124929d76937d0572a0b3a4a11: Status 404 returned error can't find the container with id 760b012ffc49a0080251a610aedae606a80ee5124929d76937d0572a0b3a4a11
Oct 30 21:33:46.895945 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:46.895904    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-g27zs" event={"ID":"a724a1c0-c643-49ff-a555-5bddec4bfacb","Type":"ContainerStarted","Data":"e08bf987192d15e57eca415b19ea759d622bac7434a523715944e31703bef122"}
Oct 30 21:33:46.897738 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:46.897714    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-588c778845-rvqdr" event={"ID":"8717a682-defa-4983-8106-b075a1d20dd3","Type":"ContainerStarted","Data":"84f46e762d869b6ed29b00a6702958aa646f9444617ae7a40fbeb8bf0eb3c3ea"}
Oct 30 21:33:46.897874 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:46.897759    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-588c778845-rvqdr" event={"ID":"8717a682-defa-4983-8106-b075a1d20dd3","Type":"ContainerStarted","Data":"760b012ffc49a0080251a610aedae606a80ee5124929d76937d0572a0b3a4a11"}
Oct 30 21:33:46.897874 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:46.897804    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-588c778845-rvqdr"
Oct 30 21:33:46.899174 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:46.899151    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-2lvn8" event={"ID":"1c873b45-e105-43e0-a528-9b327f68114c","Type":"ContainerStarted","Data":"8902990444cd3410b7c1c0f423fedfa095061c163dcdad217f4c090a6c1aedde"}
Oct 30 21:33:47.138999 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:47.138950    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-588c778845-rvqdr" podStartSLOduration=2.13893503 podStartE2EDuration="2.13893503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:47.138228857 +0000 UTC m=+21.612732837" watchObservedRunningTime="2026-10-30 21:33:47.13893503 +0000 UTC m=+21.613439003"
Oct 30 21:33:47.174373 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:33:47.174261    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:33:47.906855 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:47.906817    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hmplx" event={"ID":"7589a4df-2be1-47b1-836f-09ba72813d82","Type":"ContainerStarted","Data":"701769aedba662986852560fb977ca08ea45a9a40d05e0a8f1554cad62ec62fd"}
Oct 30 21:33:48.020329 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:48.020271    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hmplx" podStartSLOduration=1.428597911 podStartE2EDuration="3.020255468s" totalImagesPullingTime="1.591657557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:33:48.020182826 +0000 UTC m=+22.494686807" watchObservedRunningTime="2026-10-30 21:33:48.020255468 +0000 UTC m=+22.494759445"
Oct 30 21:33:48.912267 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:48.912232    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hmplx"
Oct 30 21:33:48.918723 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:48.918702    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-hmplx"
Oct 30 21:33:53.882740 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:53.882707    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:33:55.240439 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:55.240411    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:33:55.289996 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:55.289967    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-spx9v"
Oct 30 21:33:56.951798 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:56.951766    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bczf4"]
Oct 30 21:33:56.980368 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:56.980315    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bczf4"
Oct 30 21:33:57.035730 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.035701    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 21:33:57.085965 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.085208    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 21:33:57.085965 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.085237    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 21:33:57.085965 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.085297    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 21:33:57.085965 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.085319    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-l2n6f\""
Oct 30 21:33:57.123673 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.123641    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50d6c5b6-5d77-4098-8635-b4f38cae132f-metrics-client-ca\") pod \"node-exporter-bczf4\" (UID: \"50d6c5b6-5d77-4098-8635-b4f38cae132f\") " pod="openshift-monitoring/node-exporter-bczf4"
Oct 30 21:33:57.123864 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.123698    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/50d6c5b6-5d77-4098-8635-b4f38cae132f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bczf4\" (UID: \"50d6c5b6-5d77-4098-8635-b4f38cae132f\") " pod="openshift-monitoring/node-exporter-bczf4"
Oct 30 21:33:57.123864 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.123739    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/50d6c5b6-5d77-4098-8635-b4f38cae132f-node-exporter-textfile\") pod \"node-exporter-bczf4\" (UID: \"50d6c5b6-5d77-4098-8635-b4f38cae132f\") " pod="openshift-monitoring/node-exporter-bczf4"
Oct 30 21:33:57.123864 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.123768    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/50d6c5b6-5d77-4098-8635-b4f38cae132f-node-exporter-tls\") pod \"node-exporter-bczf4\" (UID: \"50d6c5b6-5d77-4098-8635-b4f38cae132f\") " pod="openshift-monitoring/node-exporter-bczf4"
Oct 30 21:33:57.123864 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.123799    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/50d6c5b6-5d77-4098-8635-b4f38cae132f-node-exporter-wtmp\") pod \"node-exporter-bczf4\" (UID: \"50d6c5b6-5d77-4098-8635-b4f38cae132f\") " pod="openshift-monitoring/node-exporter-bczf4"
Oct 30 21:33:57.123864 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.123846    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50d6c5b6-5d77-4098-8635-b4f38cae132f-sys\") pod \"node-exporter-bczf4\" (UID: \"50d6c5b6-5d77-4098-8635-b4f38cae132f\") " pod="openshift-monitoring/node-exporter-bczf4"
Oct 30 21:33:57.124074 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.123870    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/50d6c5b6-5d77-4098-8635-b4f38cae132f-node-exporter-accelerators-collector-config\") pod \"node-exporter-bczf4\" (UID: \"50d6c5b6-5d77-4098-8635-b4f38cae132f\") " pod="openshift-monitoring/node-exporter-bczf4"
Oct 30 21:33:57.124074 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.123897    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt28w\" (UniqueName: \"kubernetes.io/projected/50d6c5b6-5d77-4098-8635-b4f38cae132f-kube-api-access-kt28w\") pod \"node-exporter-bczf4\" (UID: \"50d6c5b6-5d77-4098-8635-b4f38cae132f\") " pod="openshift-monitoring/node-exporter-bczf4"
Oct 30 21:33:57.124074 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.123923    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/50d6c5b6-5d77-4098-8635-b4f38cae132f-root\") pod \"node-exporter-bczf4\" (UID: \"50d6c5b6-5d77-4098-8635-b4f38cae132f\") " pod="openshift-monitoring/node-exporter-bczf4"
Oct 30 21:33:57.224847 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.224776    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50d6c5b6-5d77-4098-8635-b4f38cae132f-sys\") pod \"node-exporter-bczf4\" (UID: \"50d6c5b6-5d77-4098-8635-b4f38cae132f\") " pod="openshift-monitoring/node-exporter-bczf4"
Oct 30 21:33:57.224847 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.224816    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/50d6c5b6-5d77-4098-8635-b4f38cae132f-node-exporter-accelerators-collector-config\") pod \"node-exporter-bczf4\" (UID: \"50d6c5b6-5d77-4098-8635-b4f38cae132f\") " pod="openshift-monitoring/node-exporter-bczf4"
Oct 30 21:33:57.224847 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.224844    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt28w\" (UniqueName: \"kubernetes.io/projected/50d6c5b6-5d77-4098-8635-b4f38cae132f-kube-api-access-kt28w\") pod \"node-exporter-bczf4\" (UID: \"50d6c5b6-5d77-4098-8635-b4f38cae132f\") " pod="openshift-monitoring/node-exporter-bczf4"
Oct 30 21:33:57.225078 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.224905    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/50d6c5b6-5d77-4098-8635-b4f38cae132f-root\") pod \"node-exporter-bczf4\" (UID: \"50d6c5b6-5d77-4098-8635-b4f38cae132f\") " pod="openshift-monitoring/node-exporter-bczf4"
Oct 30 21:33:57.225078 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.224962    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50d6c5b6-5d77-4098-8635-b4f38cae132f-metrics-client-ca\") pod \"node-exporter-bczf4\" (UID: \"50d6c5b6-5d77-4098-8635-b4f38cae132f\") " pod="openshift-monitoring/node-exporter-bczf4"
Oct 30 21:33:57.225078 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.225000    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/50d6c5b6-5d77-4098-8635-b4f38cae132f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bczf4\" (UID: \"50d6c5b6-5d77-4098-8635-b4f38cae132f\") " pod="openshift-monitoring/node-exporter-bczf4"
Oct 30 21:33:57.225078 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.225032    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/50d6c5b6-5d77-4098-8635-b4f38cae132f-node-exporter-textfile\") pod \"node-exporter-bczf4\" (UID: \"50d6c5b6-5d77-4098-8635-b4f38cae132f\") " pod="openshift-monitoring/node-exporter-bczf4"
Oct 30 21:33:57.225078 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.225049    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/50d6c5b6-5d77-4098-8635-b4f38cae132f-node-exporter-tls\") pod \"node-exporter-bczf4\" (UID: \"50d6c5b6-5d77-4098-8635-b4f38cae132f\") " pod="openshift-monitoring/node-exporter-bczf4"
Oct 30 21:33:57.225078 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.225072    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/50d6c5b6-5d77-4098-8635-b4f38cae132f-node-exporter-wtmp\") pod \"node-exporter-bczf4\" (UID: \"50d6c5b6-5d77-4098-8635-b4f38cae132f\") " pod="openshift-monitoring/node-exporter-bczf4"
Oct 30 21:33:57.225285 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.225212    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/50d6c5b6-5d77-4098-8635-b4f38cae132f-node-exporter-wtmp\") pod \"node-exporter-bczf4\" (UID: \"50d6c5b6-5d77-4098-8635-b4f38cae132f\") " pod="openshift-monitoring/node-exporter-bczf4"
Oct 30 21:33:57.225431 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.225413    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/50d6c5b6-5d77-4098-8635-b4f38cae132f-root\") pod \"node-exporter-bczf4\" (UID: \"50d6c5b6-5d77-4098-8635-b4f38cae132f\") " pod="openshift-monitoring/node-exporter-bczf4"
Oct 30 21:33:57.225501 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.225441    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/50d6c5b6-5d77-4098-8635-b4f38cae132f-node-exporter-textfile\") pod \"node-exporter-bczf4\" (UID: \"50d6c5b6-5d77-4098-8635-b4f38cae132f\") " pod="openshift-monitoring/node-exporter-bczf4"
Oct 30 21:33:57.225501 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.225463    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/50d6c5b6-5d77-4098-8635-b4f38cae132f-node-exporter-accelerators-collector-config\") pod \"node-exporter-bczf4\" (UID: \"50d6c5b6-5d77-4098-8635-b4f38cae132f\") " pod="openshift-monitoring/node-exporter-bczf4"
Oct 30 21:33:57.225603 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.225523    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/50d6c5b6-5d77-4098-8635-b4f38cae132f-sys\") pod \"node-exporter-bczf4\" (UID: \"50d6c5b6-5d77-4098-8635-b4f38cae132f\") " pod="openshift-monitoring/node-exporter-bczf4"
Oct 30 21:33:57.225840 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.225818    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50d6c5b6-5d77-4098-8635-b4f38cae132f-metrics-client-ca\") pod \"node-exporter-bczf4\" (UID: \"50d6c5b6-5d77-4098-8635-b4f38cae132f\") " pod="openshift-monitoring/node-exporter-bczf4"
Oct 30 21:33:57.227907 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.227860    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/50d6c5b6-5d77-4098-8635-b4f38cae132f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bczf4\" (UID: \"50d6c5b6-5d77-4098-8635-b4f38cae132f\") " pod="openshift-monitoring/node-exporter-bczf4"
Oct 30 21:33:57.232025 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.231979    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/50d6c5b6-5d77-4098-8635-b4f38cae132f-node-exporter-tls\") pod \"node-exporter-bczf4\" (UID: \"50d6c5b6-5d77-4098-8635-b4f38cae132f\") " pod="openshift-monitoring/node-exporter-bczf4"
Oct 30 21:33:57.281500 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.281475    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt28w\" (UniqueName: \"kubernetes.io/projected/50d6c5b6-5d77-4098-8635-b4f38cae132f-kube-api-access-kt28w\") pod \"node-exporter-bczf4\" (UID: \"50d6c5b6-5d77-4098-8635-b4f38cae132f\") " pod="openshift-monitoring/node-exporter-bczf4"
Oct 30 21:33:57.348717 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.348686    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bczf4"
Oct 30 21:33:57.389768 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:33:57.389736    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50d6c5b6_5d77_4098_8635_b4f38cae132f.slice/crio-bf65959de391f2d0b870c4c29e525f55363fda0a4f727785c46721d99b430a92 WatchSource:0}: Error finding container bf65959de391f2d0b870c4c29e525f55363fda0a4f727785c46721d99b430a92: Status 404 returned error can't find the container with id bf65959de391f2d0b870c4c29e525f55363fda0a4f727785c46721d99b430a92
Oct 30 21:33:57.956776 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:57.956733    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bczf4" event={"ID":"50d6c5b6-5d77-4098-8635-b4f38cae132f","Type":"ContainerStarted","Data":"bf65959de391f2d0b870c4c29e525f55363fda0a4f727785c46721d99b430a92"}
Oct 30 21:33:58.964467 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:58.964433    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bczf4" event={"ID":"50d6c5b6-5d77-4098-8635-b4f38cae132f","Type":"ContainerStarted","Data":"2cf515219beca320f6010a0b785bba1f9e2f3ce3d6605fbb2475152465b9e8be"}
Oct 30 21:33:59.972115 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:59.972071    5087 generic.go:356] "Generic (PLEG): container finished" podID="50d6c5b6-5d77-4098-8635-b4f38cae132f" containerID="2cf515219beca320f6010a0b785bba1f9e2f3ce3d6605fbb2475152465b9e8be" exitCode=0
Oct 30 21:33:59.972614 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:33:59.972183    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bczf4" event={"ID":"50d6c5b6-5d77-4098-8635-b4f38cae132f","Type":"ContainerDied","Data":"2cf515219beca320f6010a0b785bba1f9e2f3ce3d6605fbb2475152465b9e8be"}
Oct 30 21:34:00.980934 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:00.980895    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bczf4" event={"ID":"50d6c5b6-5d77-4098-8635-b4f38cae132f","Type":"ContainerStarted","Data":"827474fc661d395fbffa329b12a90ca979eaee80a81eb665c637136b8af2ac0d"}
Oct 30 21:34:00.981501 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:00.980967    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bczf4" event={"ID":"50d6c5b6-5d77-4098-8635-b4f38cae132f","Type":"ContainerStarted","Data":"c11850ebcfc606007a5af88c4a6dae3ace66837f1ece390c41297a023cbc5b8b"}
Oct 30 21:34:01.096360 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:01.096306    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bczf4" podStartSLOduration=3.552117277 podStartE2EDuration="5.096291284s" totalImagesPullingTime="1.456780862s" totalInitContainerRuntime="87.393145ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:33:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:01.095414455 +0000 UTC m=+35.569918434" watchObservedRunningTime="2026-10-30 21:34:01.096291284 +0000 UTC m=+35.570795267"
Oct 30 21:34:02.995577 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:02.995538    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-2lvn8" event={"ID":"1c873b45-e105-43e0-a528-9b327f68114c","Type":"ContainerStarted","Data":"f0c851fe2e15956450c2b925b27a5fab7a2cb26588bb15ff609f3c700b8dbcc9"}
Oct 30 21:34:02.996055 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:02.995809    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-2lvn8"
Oct 30 21:34:03.008756 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:03.008729    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-2lvn8"
Oct 30 21:34:03.207974 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:03.207911    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-2lvn8" podStartSLOduration=1.812159834 podStartE2EDuration="18.207890145s" totalImagesPullingTime="16.395730311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:33:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:34:03.157938083 +0000 UTC m=+37.632442062" watchObservedRunningTime="2026-10-30 21:34:03.207890145 +0000 UTC m=+37.682394102"
Oct 30 21:34:07.911989 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:07.911908    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-588c778845-rvqdr"
Oct 30 21:34:08.137713 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:08.137667    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74b44b76d6-l7p76"]
Oct 30 21:34:10.257319 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:10.257254    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7fc9466475-jz9vh" podUID="42dc1fb7-b6e0-40d2-bf53-20079815b12d" containerName="registry" containerID="cri-o://df6b3632666c81970dcfd5816025ba4d7e899974efd8e6fd819e90683624a69e" gracePeriod=30
Oct 30 21:34:10.565804 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:10.565778    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:34:10.656515 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:10.656481    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-bound-sa-token\") pod \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") "
Oct 30 21:34:10.656696 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:10.656540    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-certificates\") pod \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") "
Oct 30 21:34:10.656696 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:10.656571    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42dc1fb7-b6e0-40d2-bf53-20079815b12d-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42dc1fb7-b6e0-40d2-bf53-20079815b12d-installation-pull-secrets\") pod \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") "
Oct 30 21:34:10.656696 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:10.656601    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-kube-api-access-77rbg\" (UniqueName: \"kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-kube-api-access-77rbg\") pod \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") "
Oct 30 21:34:10.656696 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:10.656661    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42dc1fb7-b6e0-40d2-bf53-20079815b12d-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42dc1fb7-b6e0-40d2-bf53-20079815b12d-trusted-ca\") pod \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") "
Oct 30 21:34:10.656952 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:10.656708    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42dc1fb7-b6e0-40d2-bf53-20079815b12d-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42dc1fb7-b6e0-40d2-bf53-20079815b12d-ca-trust-extracted\") pod \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") "
Oct 30 21:34:10.656952 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:10.656751    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42dc1fb7-b6e0-40d2-bf53-20079815b12d-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42dc1fb7-b6e0-40d2-bf53-20079815b12d-image-registry-private-configuration\") pod \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") "
Oct 30 21:34:10.656952 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:10.656808    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-tls\" (UniqueName: \"kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-tls\") pod \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\" (UID: \"42dc1fb7-b6e0-40d2-bf53-20079815b12d\") "
Oct 30 21:34:10.657111 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:10.656950    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-certificates" pod "42dc1fb7-b6e0-40d2-bf53-20079815b12d" (UID: "42dc1fb7-b6e0-40d2-bf53-20079815b12d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:10.657111 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:10.657084    5087 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-certificates\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:34:10.657608 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:10.657324    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42dc1fb7-b6e0-40d2-bf53-20079815b12d-trusted-ca" pod "42dc1fb7-b6e0-40d2-bf53-20079815b12d" (UID: "42dc1fb7-b6e0-40d2-bf53-20079815b12d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:10.659942 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:10.659904    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42dc1fb7-b6e0-40d2-bf53-20079815b12d-installation-pull-secrets" pod "42dc1fb7-b6e0-40d2-bf53-20079815b12d" (UID: "42dc1fb7-b6e0-40d2-bf53-20079815b12d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:34:10.660459 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:10.660429    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42dc1fb7-b6e0-40d2-bf53-20079815b12d-image-registry-private-configuration" pod "42dc1fb7-b6e0-40d2-bf53-20079815b12d" (UID: "42dc1fb7-b6e0-40d2-bf53-20079815b12d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:34:10.660836 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:10.660608    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-kube-api-access-77rbg" pod "42dc1fb7-b6e0-40d2-bf53-20079815b12d" (UID: "42dc1fb7-b6e0-40d2-bf53-20079815b12d"). InnerVolumeSpecName "kube-api-access-77rbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:10.660836 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:10.660694    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-tls" pod "42dc1fb7-b6e0-40d2-bf53-20079815b12d" (UID: "42dc1fb7-b6e0-40d2-bf53-20079815b12d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:10.660950 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:10.660891    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-bound-sa-token" pod "42dc1fb7-b6e0-40d2-bf53-20079815b12d" (UID: "42dc1fb7-b6e0-40d2-bf53-20079815b12d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:10.666682 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:10.666655    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42dc1fb7-b6e0-40d2-bf53-20079815b12d-ca-trust-extracted" pod "42dc1fb7-b6e0-40d2-bf53-20079815b12d" (UID: "42dc1fb7-b6e0-40d2-bf53-20079815b12d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:10.758232 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:10.758195    5087 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42dc1fb7-b6e0-40d2-bf53-20079815b12d-installation-pull-secrets\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:34:10.758232 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:10.758226    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77rbg\" (UniqueName: \"kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-kube-api-access-77rbg\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:34:10.758232 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:10.758237    5087 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42dc1fb7-b6e0-40d2-bf53-20079815b12d-trusted-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:34:10.758540 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:10.758246    5087 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42dc1fb7-b6e0-40d2-bf53-20079815b12d-ca-trust-extracted\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:34:10.758540 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:10.758259    5087 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42dc1fb7-b6e0-40d2-bf53-20079815b12d-image-registry-private-configuration\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:34:10.758540 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:10.758274    5087 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-registry-tls\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:34:10.758540 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:10.758287    5087 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42dc1fb7-b6e0-40d2-bf53-20079815b12d-bound-sa-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:34:11.041259 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:11.041221    5087 generic.go:356] "Generic (PLEG): container finished" podID="42dc1fb7-b6e0-40d2-bf53-20079815b12d" containerID="df6b3632666c81970dcfd5816025ba4d7e899974efd8e6fd819e90683624a69e" exitCode=0
Oct 30 21:34:11.041476 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:11.041277    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7fc9466475-jz9vh"
Oct 30 21:34:11.041476 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:11.041311    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fc9466475-jz9vh" event={"ID":"42dc1fb7-b6e0-40d2-bf53-20079815b12d","Type":"ContainerDied","Data":"df6b3632666c81970dcfd5816025ba4d7e899974efd8e6fd819e90683624a69e"}
Oct 30 21:34:11.041476 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:11.041402    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7fc9466475-jz9vh" event={"ID":"42dc1fb7-b6e0-40d2-bf53-20079815b12d","Type":"ContainerDied","Data":"993ef86e4edeac393959e3a25b8b27a6be355fbe898750ae4f421775c4c5a146"}
Oct 30 21:34:11.041476 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:11.041433    5087 scope.go:122] "RemoveContainer" containerID="df6b3632666c81970dcfd5816025ba4d7e899974efd8e6fd819e90683624a69e"
Oct 30 21:34:11.052802 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:11.052775    5087 scope.go:122] "RemoveContainer" containerID="df6b3632666c81970dcfd5816025ba4d7e899974efd8e6fd819e90683624a69e"
Oct 30 21:34:11.053124 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:34:11.053101    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df6b3632666c81970dcfd5816025ba4d7e899974efd8e6fd819e90683624a69e\": container with ID starting with df6b3632666c81970dcfd5816025ba4d7e899974efd8e6fd819e90683624a69e not found: ID does not exist" containerID="df6b3632666c81970dcfd5816025ba4d7e899974efd8e6fd819e90683624a69e"
Oct 30 21:34:11.053216 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:11.053152    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df6b3632666c81970dcfd5816025ba4d7e899974efd8e6fd819e90683624a69e"} err="failed to get container status \"df6b3632666c81970dcfd5816025ba4d7e899974efd8e6fd819e90683624a69e\": rpc error: code = NotFound desc = could not find container \"df6b3632666c81970dcfd5816025ba4d7e899974efd8e6fd819e90683624a69e\": container with ID starting with df6b3632666c81970dcfd5816025ba4d7e899974efd8e6fd819e90683624a69e not found: ID does not exist"
Oct 30 21:34:11.206645 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:11.206612    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7fc9466475-jz9vh"]
Oct 30 21:34:11.255592 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:11.255560    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7fc9466475-jz9vh"]
Oct 30 21:34:11.707312 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:11.707284    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42dc1fb7-b6e0-40d2-bf53-20079815b12d" path="/var/lib/kubelet/pods/42dc1fb7-b6e0-40d2-bf53-20079815b12d/volumes"
Oct 30 21:34:17.166101 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:34:17.166062    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:34:17.914855 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:17.914825    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-4sgw4-zcs8v"]
Oct 30 21:34:17.915076 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:17.915053    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-4sgw4-zcs8v" podUID="5278b436-9321-4856-b9b0-ce9eb2596775" containerName="insights-gathering" containerID="cri-o://6c85c9004b2949eb998a21c5af46307116a0c21a749672fc775ee8df1cc0f88a" gracePeriod=30
Oct 30 21:34:18.073141 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:18.073107    5087 generic.go:356] "Generic (PLEG): container finished" podID="5278b436-9321-4856-b9b0-ce9eb2596775" containerID="6c85c9004b2949eb998a21c5af46307116a0c21a749672fc775ee8df1cc0f88a" exitCode=2
Oct 30 21:34:18.073260 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:18.073178    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4sgw4-zcs8v" event={"ID":"5278b436-9321-4856-b9b0-ce9eb2596775","Type":"ContainerDied","Data":"6c85c9004b2949eb998a21c5af46307116a0c21a749672fc775ee8df1cc0f88a"}
Oct 30 21:34:18.223594 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:18.223567    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4sgw4-zcs8v"
Oct 30 21:34:18.321857 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:18.321823    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5278b436-9321-4856-b9b0-ce9eb2596775-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5278b436-9321-4856-b9b0-ce9eb2596775-service-ca-bundle\") pod \"5278b436-9321-4856-b9b0-ce9eb2596775\" (UID: \"5278b436-9321-4856-b9b0-ce9eb2596775\") "
Oct 30 21:34:18.322011 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:18.321892    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5278b436-9321-4856-b9b0-ce9eb2596775-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5278b436-9321-4856-b9b0-ce9eb2596775-archives-path\") pod \"5278b436-9321-4856-b9b0-ce9eb2596775\" (UID: \"5278b436-9321-4856-b9b0-ce9eb2596775\") "
Oct 30 21:34:18.322011 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:18.321922    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5278b436-9321-4856-b9b0-ce9eb2596775-kube-api-access-48g5f\" (UniqueName: \"kubernetes.io/projected/5278b436-9321-4856-b9b0-ce9eb2596775-kube-api-access-48g5f\") pod \"5278b436-9321-4856-b9b0-ce9eb2596775\" (UID: \"5278b436-9321-4856-b9b0-ce9eb2596775\") "
Oct 30 21:34:18.322011 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:18.321965    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5278b436-9321-4856-b9b0-ce9eb2596775-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5278b436-9321-4856-b9b0-ce9eb2596775-trusted-ca-bundle\") pod \"5278b436-9321-4856-b9b0-ce9eb2596775\" (UID: \"5278b436-9321-4856-b9b0-ce9eb2596775\") "
Oct 30 21:34:18.322212 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:18.322181    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5278b436-9321-4856-b9b0-ce9eb2596775-service-ca-bundle" pod "5278b436-9321-4856-b9b0-ce9eb2596775" (UID: "5278b436-9321-4856-b9b0-ce9eb2596775"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:18.322289 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:18.322254    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5278b436-9321-4856-b9b0-ce9eb2596775-archives-path" pod "5278b436-9321-4856-b9b0-ce9eb2596775" (UID: "5278b436-9321-4856-b9b0-ce9eb2596775"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:18.322433 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:18.322410    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5278b436-9321-4856-b9b0-ce9eb2596775-trusted-ca-bundle" pod "5278b436-9321-4856-b9b0-ce9eb2596775" (UID: "5278b436-9321-4856-b9b0-ce9eb2596775"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:18.324281 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:18.324261    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5278b436-9321-4856-b9b0-ce9eb2596775-kube-api-access-48g5f" pod "5278b436-9321-4856-b9b0-ce9eb2596775" (UID: "5278b436-9321-4856-b9b0-ce9eb2596775"). InnerVolumeSpecName "kube-api-access-48g5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:18.422674 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:18.422638    5087 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5278b436-9321-4856-b9b0-ce9eb2596775-service-ca-bundle\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:34:18.422674 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:18.422669    5087 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5278b436-9321-4856-b9b0-ce9eb2596775-archives-path\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:34:18.422935 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:18.422682    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48g5f\" (UniqueName: \"kubernetes.io/projected/5278b436-9321-4856-b9b0-ce9eb2596775-kube-api-access-48g5f\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:34:18.422935 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:18.422697    5087 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5278b436-9321-4856-b9b0-ce9eb2596775-trusted-ca-bundle\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:34:19.078165 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:19.078130    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4sgw4-zcs8v"
Oct 30 21:34:19.078165 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:19.078146    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4sgw4-zcs8v" event={"ID":"5278b436-9321-4856-b9b0-ce9eb2596775","Type":"ContainerDied","Data":"1e778827f2aa19da409b22ecbb0cc7a80f0e711817a34162237798e464a6c01e"}
Oct 30 21:34:19.078556 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:19.078200    5087 scope.go:122] "RemoveContainer" containerID="6c85c9004b2949eb998a21c5af46307116a0c21a749672fc775ee8df1cc0f88a"
Oct 30 21:34:19.239575 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:19.239540    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-4sgw4-zcs8v"]
Oct 30 21:34:19.242542 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:19.242516    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-4sgw4-zcs8v"]
Oct 30 21:34:19.706966 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:19.706936    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5278b436-9321-4856-b9b0-ce9eb2596775" path="/var/lib/kubelet/pods/5278b436-9321-4856-b9b0-ce9eb2596775/volumes"
Oct 30 21:34:33.161987 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:33.161922    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-74b44b76d6-l7p76" podUID="392cc7ef-7cd2-4090-a246-0347d5af7bbf" containerName="registry" containerID="cri-o://3cad43653aef065ab53061fc1fa622173077ffa91fa66e07fc1ebd44ac2a72c9" gracePeriod=30
Oct 30 21:34:33.449681 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:33.449658    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:34:33.540549 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:33.540513    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/392cc7ef-7cd2-4090-a246-0347d5af7bbf-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/392cc7ef-7cd2-4090-a246-0347d5af7bbf-installation-pull-secrets\") pod \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") "
Oct 30 21:34:33.540727 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:33.540564    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-kube-api-access-f8z8h\" (UniqueName: \"kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-kube-api-access-f8z8h\") pod \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") "
Oct 30 21:34:33.540727 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:33.540596    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/392cc7ef-7cd2-4090-a246-0347d5af7bbf-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/392cc7ef-7cd2-4090-a246-0347d5af7bbf-image-registry-private-configuration\") pod \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") "
Oct 30 21:34:33.540727 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:33.540627    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/392cc7ef-7cd2-4090-a246-0347d5af7bbf-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/392cc7ef-7cd2-4090-a246-0347d5af7bbf-trusted-ca\") pod \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") "
Oct 30 21:34:33.540727 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:33.540659    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-tls\" (UniqueName: \"kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-tls\") pod \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") "
Oct 30 21:34:33.540727 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:33.540714    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-bound-sa-token\") pod \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") "
Oct 30 21:34:33.541002 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:33.540771    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-certificates\") pod \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") "
Oct 30 21:34:33.541226 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:33.541164    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-certificates" pod "392cc7ef-7cd2-4090-a246-0347d5af7bbf" (UID: "392cc7ef-7cd2-4090-a246-0347d5af7bbf"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:33.541366 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:33.541291    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/392cc7ef-7cd2-4090-a246-0347d5af7bbf-trusted-ca" pod "392cc7ef-7cd2-4090-a246-0347d5af7bbf" (UID: "392cc7ef-7cd2-4090-a246-0347d5af7bbf"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:34:33.541677 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:33.541649    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/392cc7ef-7cd2-4090-a246-0347d5af7bbf-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/392cc7ef-7cd2-4090-a246-0347d5af7bbf-ca-trust-extracted\") pod \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\" (UID: \"392cc7ef-7cd2-4090-a246-0347d5af7bbf\") "
Oct 30 21:34:33.541936 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:33.541915    5087 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/392cc7ef-7cd2-4090-a246-0347d5af7bbf-trusted-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:34:33.542032 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:33.541943    5087 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-certificates\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:34:33.543250 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:33.543224    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/392cc7ef-7cd2-4090-a246-0347d5af7bbf-image-registry-private-configuration" pod "392cc7ef-7cd2-4090-a246-0347d5af7bbf" (UID: "392cc7ef-7cd2-4090-a246-0347d5af7bbf"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:34:33.543364 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:33.543328    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-kube-api-access-f8z8h" pod "392cc7ef-7cd2-4090-a246-0347d5af7bbf" (UID: "392cc7ef-7cd2-4090-a246-0347d5af7bbf"). InnerVolumeSpecName "kube-api-access-f8z8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:33.543627 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:33.543579    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/392cc7ef-7cd2-4090-a246-0347d5af7bbf-installation-pull-secrets" pod "392cc7ef-7cd2-4090-a246-0347d5af7bbf" (UID: "392cc7ef-7cd2-4090-a246-0347d5af7bbf"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:34:33.543724 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:33.543652    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-bound-sa-token" pod "392cc7ef-7cd2-4090-a246-0347d5af7bbf" (UID: "392cc7ef-7cd2-4090-a246-0347d5af7bbf"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:33.543724 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:33.543663    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-tls" pod "392cc7ef-7cd2-4090-a246-0347d5af7bbf" (UID: "392cc7ef-7cd2-4090-a246-0347d5af7bbf"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:34:33.548258 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:33.548239    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/392cc7ef-7cd2-4090-a246-0347d5af7bbf-ca-trust-extracted" pod "392cc7ef-7cd2-4090-a246-0347d5af7bbf" (UID: "392cc7ef-7cd2-4090-a246-0347d5af7bbf"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:34:33.642671 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:33.642640    5087 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/392cc7ef-7cd2-4090-a246-0347d5af7bbf-ca-trust-extracted\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:34:33.642671 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:33.642665    5087 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/392cc7ef-7cd2-4090-a246-0347d5af7bbf-installation-pull-secrets\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:34:33.642671 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:33.642675    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8z8h\" (UniqueName: \"kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-kube-api-access-f8z8h\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:34:33.642874 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:33.642685    5087 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/392cc7ef-7cd2-4090-a246-0347d5af7bbf-image-registry-private-configuration\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:34:33.642874 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:33.642695    5087 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-registry-tls\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:34:33.642874 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:33.642704    5087 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/392cc7ef-7cd2-4090-a246-0347d5af7bbf-bound-sa-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:34:34.144017 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:34.143982    5087 generic.go:356] "Generic (PLEG): container finished" podID="392cc7ef-7cd2-4090-a246-0347d5af7bbf" containerID="3cad43653aef065ab53061fc1fa622173077ffa91fa66e07fc1ebd44ac2a72c9" exitCode=0
Oct 30 21:34:34.144212 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:34.144023    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74b44b76d6-l7p76"
Oct 30 21:34:34.144212 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:34.144051    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74b44b76d6-l7p76" event={"ID":"392cc7ef-7cd2-4090-a246-0347d5af7bbf","Type":"ContainerDied","Data":"3cad43653aef065ab53061fc1fa622173077ffa91fa66e07fc1ebd44ac2a72c9"}
Oct 30 21:34:34.144212 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:34.144104    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74b44b76d6-l7p76" event={"ID":"392cc7ef-7cd2-4090-a246-0347d5af7bbf","Type":"ContainerDied","Data":"8286f2dee2e87a47a7765219547a44effd40c7e5c6f940eebf1500e67a0cbdfe"}
Oct 30 21:34:34.144212 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:34.144131    5087 scope.go:122] "RemoveContainer" containerID="3cad43653aef065ab53061fc1fa622173077ffa91fa66e07fc1ebd44ac2a72c9"
Oct 30 21:34:34.152326 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:34.152309    5087 scope.go:122] "RemoveContainer" containerID="3cad43653aef065ab53061fc1fa622173077ffa91fa66e07fc1ebd44ac2a72c9"
Oct 30 21:34:34.152607 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:34:34.152582    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cad43653aef065ab53061fc1fa622173077ffa91fa66e07fc1ebd44ac2a72c9\": container with ID starting with 3cad43653aef065ab53061fc1fa622173077ffa91fa66e07fc1ebd44ac2a72c9 not found: ID does not exist" containerID="3cad43653aef065ab53061fc1fa622173077ffa91fa66e07fc1ebd44ac2a72c9"
Oct 30 21:34:34.152707 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:34.152621    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cad43653aef065ab53061fc1fa622173077ffa91fa66e07fc1ebd44ac2a72c9"} err="failed to get container status \"3cad43653aef065ab53061fc1fa622173077ffa91fa66e07fc1ebd44ac2a72c9\": rpc error: code = NotFound desc = could not find container \"3cad43653aef065ab53061fc1fa622173077ffa91fa66e07fc1ebd44ac2a72c9\": container with ID starting with 3cad43653aef065ab53061fc1fa622173077ffa91fa66e07fc1ebd44ac2a72c9 not found: ID does not exist"
Oct 30 21:34:34.355293 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:34.355256    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74b44b76d6-l7p76"]
Oct 30 21:34:34.405114 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:34.405042    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-74b44b76d6-l7p76"]
Oct 30 21:34:35.706703 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:35.706672    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="392cc7ef-7cd2-4090-a246-0347d5af7bbf" path="/var/lib/kubelet/pods/392cc7ef-7cd2-4090-a246-0347d5af7bbf/volumes"
Oct 30 21:34:47.165990 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:34:47.165948    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:34:51.210585 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:51.210552    5087 generic.go:356] "Generic (PLEG): container finished" podID="f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef" containerID="3b5f8b210727fb7582c0106909c9950cdf0a9bf1eb67d2a632d3ad5903943052" exitCode=0
Oct 30 21:34:51.210962 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:51.210637    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bn8fx" event={"ID":"f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef","Type":"ContainerDied","Data":"3b5f8b210727fb7582c0106909c9950cdf0a9bf1eb67d2a632d3ad5903943052"}
Oct 30 21:34:51.210962 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:51.210955    5087 scope.go:122] "RemoveContainer" containerID="3b5f8b210727fb7582c0106909c9950cdf0a9bf1eb67d2a632d3ad5903943052"
Oct 30 21:34:51.212295 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:51.212266    5087 generic.go:356] "Generic (PLEG): container finished" podID="25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245" containerID="eee1afe06d79357285df9d6fe0e1ca4f382b777055128461b08ab6b0da5ba34a" exitCode=0
Oct 30 21:34:51.212438 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:51.212330    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc" event={"ID":"25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245","Type":"ContainerDied","Data":"eee1afe06d79357285df9d6fe0e1ca4f382b777055128461b08ab6b0da5ba34a"}
Oct 30 21:34:51.212625 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:51.212609    5087 scope.go:122] "RemoveContainer" containerID="eee1afe06d79357285df9d6fe0e1ca4f382b777055128461b08ab6b0da5ba34a"
Oct 30 21:34:52.221166 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:52.221129    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-9dfgc" event={"ID":"25b8da2c-9c9d-41ab-8cf4-1a6d7f5e9245","Type":"ContainerStarted","Data":"9581382348dfb710c849243742d41ef37312349919ad632165668ba1fb085c14"}
Oct 30 21:34:52.223019 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:34:52.222996    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-bn8fx" event={"ID":"f9d73b67-bdd0-4ab1-a8b2-0395e8ff73ef","Type":"ContainerStarted","Data":"1396e9ecc2ac5b579df403e5617b8f4c3c94242c284cc91e94cc6a27923ff5fd"}
Oct 30 21:35:17.166326 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:35:17.166284    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:35:47.165502 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:35:47.165424    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:36:17.166693 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:36:17.166651    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:36:47.166340 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:36:47.166293    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:37:17.165796 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:37:17.165720    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:37:47.169729 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:37:47.169695    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:37:53.546408 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.546378    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"]
Oct 30 21:37:53.546913 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.546680    5087 cpu_manager.go:414] "containerMap: removing container" podUID="392cc7ef-7cd2-4090-a246-0347d5af7bbf" containerName="registry"
Oct 30 21:37:53.546913 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.546690    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="392cc7ef-7cd2-4090-a246-0347d5af7bbf" containerName="registry"
Oct 30 21:37:53.546913 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.546710    5087 cpu_manager.go:414] "containerMap: removing container" podUID="5278b436-9321-4856-b9b0-ce9eb2596775" containerName="insights-gathering"
Oct 30 21:37:53.546913 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.546715    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5278b436-9321-4856-b9b0-ce9eb2596775" containerName="insights-gathering"
Oct 30 21:37:53.546913 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.546734    5087 cpu_manager.go:414] "containerMap: removing container" podUID="42dc1fb7-b6e0-40d2-bf53-20079815b12d" containerName="registry"
Oct 30 21:37:53.546913 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.546739    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42dc1fb7-b6e0-40d2-bf53-20079815b12d" containerName="registry"
Oct 30 21:37:53.546913 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.546789    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="5278b436-9321-4856-b9b0-ce9eb2596775" containerName="insights-gathering"
Oct 30 21:37:53.546913 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.546798    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="42dc1fb7-b6e0-40d2-bf53-20079815b12d" containerName="registry"
Oct 30 21:37:53.546913 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.546804    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="392cc7ef-7cd2-4090-a246-0347d5af7bbf" containerName="registry"
Oct 30 21:37:53.548821 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.548802    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:53.585815 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.585778    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15eaf1f9-3376-4b99-b708-84a2d6af5d1b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s\" (UID: \"15eaf1f9-3376-4b99-b708-84a2d6af5d1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:53.585967 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.585829    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hg6r\" (UniqueName: \"kubernetes.io/projected/15eaf1f9-3376-4b99-b708-84a2d6af5d1b-kube-api-access-4hg6r\") pod \"openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s\" (UID: \"15eaf1f9-3376-4b99-b708-84a2d6af5d1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:53.585967 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.585929    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15eaf1f9-3376-4b99-b708-84a2d6af5d1b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s\" (UID: \"15eaf1f9-3376-4b99-b708-84a2d6af5d1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:53.585967 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.585964    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15eaf1f9-3376-4b99-b708-84a2d6af5d1b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s\" (UID: \"15eaf1f9-3376-4b99-b708-84a2d6af5d1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:53.586114 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.586055    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15eaf1f9-3376-4b99-b708-84a2d6af5d1b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s\" (UID: \"15eaf1f9-3376-4b99-b708-84a2d6af5d1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:53.586114 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.586091    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15eaf1f9-3376-4b99-b708-84a2d6af5d1b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s\" (UID: \"15eaf1f9-3376-4b99-b708-84a2d6af5d1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:53.586222 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.586176    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15eaf1f9-3376-4b99-b708-84a2d6af5d1b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s\" (UID: \"15eaf1f9-3376-4b99-b708-84a2d6af5d1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:53.586265 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.586224    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15eaf1f9-3376-4b99-b708-84a2d6af5d1b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s\" (UID: \"15eaf1f9-3376-4b99-b708-84a2d6af5d1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:53.586299 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.586279    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15eaf1f9-3376-4b99-b708-84a2d6af5d1b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s\" (UID: \"15eaf1f9-3376-4b99-b708-84a2d6af5d1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:53.603406 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.603382    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 21:37:53.630667 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.630649    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-mwgct\""
Oct 30 21:37:53.653384 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.653336    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"]
Oct 30 21:37:53.686837 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.686809    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15eaf1f9-3376-4b99-b708-84a2d6af5d1b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s\" (UID: \"15eaf1f9-3376-4b99-b708-84a2d6af5d1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:53.686956 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.686845    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15eaf1f9-3376-4b99-b708-84a2d6af5d1b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s\" (UID: \"15eaf1f9-3376-4b99-b708-84a2d6af5d1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:53.686956 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.686866    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15eaf1f9-3376-4b99-b708-84a2d6af5d1b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s\" (UID: \"15eaf1f9-3376-4b99-b708-84a2d6af5d1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:53.686956 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.686890    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15eaf1f9-3376-4b99-b708-84a2d6af5d1b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s\" (UID: \"15eaf1f9-3376-4b99-b708-84a2d6af5d1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:53.686956 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.686917    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hg6r\" (UniqueName: \"kubernetes.io/projected/15eaf1f9-3376-4b99-b708-84a2d6af5d1b-kube-api-access-4hg6r\") pod \"openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s\" (UID: \"15eaf1f9-3376-4b99-b708-84a2d6af5d1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:53.687167 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.686955    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15eaf1f9-3376-4b99-b708-84a2d6af5d1b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s\" (UID: \"15eaf1f9-3376-4b99-b708-84a2d6af5d1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:53.687167 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.686978    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15eaf1f9-3376-4b99-b708-84a2d6af5d1b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s\" (UID: \"15eaf1f9-3376-4b99-b708-84a2d6af5d1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:53.687167 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.687008    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15eaf1f9-3376-4b99-b708-84a2d6af5d1b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s\" (UID: \"15eaf1f9-3376-4b99-b708-84a2d6af5d1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:53.687167 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.687045    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15eaf1f9-3376-4b99-b708-84a2d6af5d1b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s\" (UID: \"15eaf1f9-3376-4b99-b708-84a2d6af5d1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:53.687385 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.687284    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15eaf1f9-3376-4b99-b708-84a2d6af5d1b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s\" (UID: \"15eaf1f9-3376-4b99-b708-84a2d6af5d1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:53.687427 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.687393    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15eaf1f9-3376-4b99-b708-84a2d6af5d1b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s\" (UID: \"15eaf1f9-3376-4b99-b708-84a2d6af5d1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:53.687490 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.687459    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15eaf1f9-3376-4b99-b708-84a2d6af5d1b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s\" (UID: \"15eaf1f9-3376-4b99-b708-84a2d6af5d1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:53.687563 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.687541    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15eaf1f9-3376-4b99-b708-84a2d6af5d1b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s\" (UID: \"15eaf1f9-3376-4b99-b708-84a2d6af5d1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:53.687747 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.687723    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15eaf1f9-3376-4b99-b708-84a2d6af5d1b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s\" (UID: \"15eaf1f9-3376-4b99-b708-84a2d6af5d1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:53.689292 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.689271    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15eaf1f9-3376-4b99-b708-84a2d6af5d1b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s\" (UID: \"15eaf1f9-3376-4b99-b708-84a2d6af5d1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:53.689836 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.689813    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15eaf1f9-3376-4b99-b708-84a2d6af5d1b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s\" (UID: \"15eaf1f9-3376-4b99-b708-84a2d6af5d1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:53.744026 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.744001    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hg6r\" (UniqueName: \"kubernetes.io/projected/15eaf1f9-3376-4b99-b708-84a2d6af5d1b-kube-api-access-4hg6r\") pod \"openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s\" (UID: \"15eaf1f9-3376-4b99-b708-84a2d6af5d1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:53.744156 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.744108    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15eaf1f9-3376-4b99-b708-84a2d6af5d1b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s\" (UID: \"15eaf1f9-3376-4b99-b708-84a2d6af5d1b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:53.860364 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:53.860281    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:54.278466 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:54.278437    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"]
Oct 30 21:37:54.282273 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:37:54.282237    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15eaf1f9_3376_4b99_b708_84a2d6af5d1b.slice/crio-c8f6e6c9abe59b10952e1ddaf3fd1725f1e603cab623adf4079c14cd640f7756 WatchSource:0}: Error finding container c8f6e6c9abe59b10952e1ddaf3fd1725f1e603cab623adf4079c14cd640f7756: Status 404 returned error can't find the container with id c8f6e6c9abe59b10952e1ddaf3fd1725f1e603cab623adf4079c14cd640f7756
Oct 30 21:37:54.933862 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:54.933830    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s" event={"ID":"15eaf1f9-3376-4b99-b708-84a2d6af5d1b","Type":"ContainerStarted","Data":"c8f6e6c9abe59b10952e1ddaf3fd1725f1e603cab623adf4079c14cd640f7756"}
Oct 30 21:37:57.872132 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:57.872101    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:37:57.872443 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:57.872168    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:37:57.872443 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:57.872216    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:37:57.948479 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:57.948434    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s" event={"ID":"15eaf1f9-3376-4b99-b708-84a2d6af5d1b","Type":"ContainerStarted","Data":"0d9141dd51aa12bb84222caee49a1b6a10838e8fc298ddb134554d12bd573949"}
Oct 30 21:37:58.095208 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:58.095153    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s" podStartSLOduration=1.507837815 podStartE2EDuration="5.095140128s" totalImagesPullingTime="3.587302313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:58.094187098 +0000 UTC m=+272.568691076" watchObservedRunningTime="2026-10-30 21:37:58.095140128 +0000 UTC m=+272.569644195"
Oct 30 21:37:58.861027 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:58.860993    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:58.865963 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:58.865936    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:58.953021 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:58.952992    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:37:58.953794 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:37:58.953771    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s"
Oct 30 21:38:17.169251 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:38:17.167211    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:38:17.631922 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:17.631887    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7c746f769b-7h4sb"]
Oct 30 21:38:17.641702 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:17.641678    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c746f769b-7h4sb"
Oct 30 21:38:17.674385 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:17.674341    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b09d9c97-c4b6-4fbf-9652-6928964f6364-tmp\") pod \"opendatahub-operator-controller-manager-7c746f769b-7h4sb\" (UID: \"b09d9c97-c4b6-4fbf-9652-6928964f6364\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c746f769b-7h4sb"
Oct 30 21:38:17.674508 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:17.674394    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2knx6\" (UniqueName: \"kubernetes.io/projected/b09d9c97-c4b6-4fbf-9652-6928964f6364-kube-api-access-2knx6\") pod \"opendatahub-operator-controller-manager-7c746f769b-7h4sb\" (UID: \"b09d9c97-c4b6-4fbf-9652-6928964f6364\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c746f769b-7h4sb"
Oct 30 21:38:17.674508 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:17.674470    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b09d9c97-c4b6-4fbf-9652-6928964f6364-manifests\") pod \"opendatahub-operator-controller-manager-7c746f769b-7h4sb\" (UID: \"b09d9c97-c4b6-4fbf-9652-6928964f6364\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c746f769b-7h4sb"
Oct 30 21:38:17.674597 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:17.674514    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b09d9c97-c4b6-4fbf-9652-6928964f6364-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c746f769b-7h4sb\" (UID: \"b09d9c97-c4b6-4fbf-9652-6928964f6364\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c746f769b-7h4sb"
Oct 30 21:38:17.674635 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:17.674610    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b09d9c97-c4b6-4fbf-9652-6928964f6364-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c746f769b-7h4sb\" (UID: \"b09d9c97-c4b6-4fbf-9652-6928964f6364\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c746f769b-7h4sb"
Oct 30 21:38:17.674671 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:17.674639    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b09d9c97-c4b6-4fbf-9652-6928964f6364-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c746f769b-7h4sb\" (UID: \"b09d9c97-c4b6-4fbf-9652-6928964f6364\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c746f769b-7h4sb"
Oct 30 21:38:17.759327 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:17.759294    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 21:38:17.759519 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:17.759388    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 21:38:17.759519 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:17.759387    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 21:38:17.759519 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:17.759388    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 21:38:17.759519 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:17.759407    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 21:38:17.759519 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:17.759432    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-xkqbq\""
Oct 30 21:38:17.761181 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:17.761161    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7c746f769b-7h4sb"]
Oct 30 21:38:17.775887 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:17.775861    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b09d9c97-c4b6-4fbf-9652-6928964f6364-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c746f769b-7h4sb\" (UID: \"b09d9c97-c4b6-4fbf-9652-6928964f6364\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c746f769b-7h4sb"
Oct 30 21:38:17.776020 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:17.775897    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b09d9c97-c4b6-4fbf-9652-6928964f6364-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c746f769b-7h4sb\" (UID: \"b09d9c97-c4b6-4fbf-9652-6928964f6364\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c746f769b-7h4sb"
Oct 30 21:38:17.776020 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:17.775938    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b09d9c97-c4b6-4fbf-9652-6928964f6364-tmp\") pod \"opendatahub-operator-controller-manager-7c746f769b-7h4sb\" (UID: \"b09d9c97-c4b6-4fbf-9652-6928964f6364\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c746f769b-7h4sb"
Oct 30 21:38:17.776020 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:17.775968    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2knx6\" (UniqueName: \"kubernetes.io/projected/b09d9c97-c4b6-4fbf-9652-6928964f6364-kube-api-access-2knx6\") pod \"opendatahub-operator-controller-manager-7c746f769b-7h4sb\" (UID: \"b09d9c97-c4b6-4fbf-9652-6928964f6364\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c746f769b-7h4sb"
Oct 30 21:38:17.776020 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:17.776009    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b09d9c97-c4b6-4fbf-9652-6928964f6364-manifests\") pod \"opendatahub-operator-controller-manager-7c746f769b-7h4sb\" (UID: \"b09d9c97-c4b6-4fbf-9652-6928964f6364\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c746f769b-7h4sb"
Oct 30 21:38:17.776234 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:17.776076    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b09d9c97-c4b6-4fbf-9652-6928964f6364-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c746f769b-7h4sb\" (UID: \"b09d9c97-c4b6-4fbf-9652-6928964f6364\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c746f769b-7h4sb"
Oct 30 21:38:17.776579 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:17.776556    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b09d9c97-c4b6-4fbf-9652-6928964f6364-manifests\") pod \"opendatahub-operator-controller-manager-7c746f769b-7h4sb\" (UID: \"b09d9c97-c4b6-4fbf-9652-6928964f6364\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c746f769b-7h4sb"
Oct 30 21:38:17.778376 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:17.778322    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b09d9c97-c4b6-4fbf-9652-6928964f6364-tmp\") pod \"opendatahub-operator-controller-manager-7c746f769b-7h4sb\" (UID: \"b09d9c97-c4b6-4fbf-9652-6928964f6364\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c746f769b-7h4sb"
Oct 30 21:38:17.778646 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:17.778622    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b09d9c97-c4b6-4fbf-9652-6928964f6364-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c746f769b-7h4sb\" (UID: \"b09d9c97-c4b6-4fbf-9652-6928964f6364\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c746f769b-7h4sb"
Oct 30 21:38:17.779008 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:17.778988    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b09d9c97-c4b6-4fbf-9652-6928964f6364-metrics-cert\") pod \"opendatahub-operator-controller-manager-7c746f769b-7h4sb\" (UID: \"b09d9c97-c4b6-4fbf-9652-6928964f6364\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c746f769b-7h4sb"
Oct 30 21:38:17.779082 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:17.779023    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b09d9c97-c4b6-4fbf-9652-6928964f6364-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c746f769b-7h4sb\" (UID: \"b09d9c97-c4b6-4fbf-9652-6928964f6364\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c746f769b-7h4sb"
Oct 30 21:38:17.834808 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:17.834784    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2knx6\" (UniqueName: \"kubernetes.io/projected/b09d9c97-c4b6-4fbf-9652-6928964f6364-kube-api-access-2knx6\") pod \"opendatahub-operator-controller-manager-7c746f769b-7h4sb\" (UID: \"b09d9c97-c4b6-4fbf-9652-6928964f6364\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c746f769b-7h4sb"
Oct 30 21:38:17.956704 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:17.956665    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c746f769b-7h4sb"
Oct 30 21:38:18.494000 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:18.493973    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7c746f769b-7h4sb"]
Oct 30 21:38:18.496216 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:38:18.496190    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb09d9c97_c4b6_4fbf_9652_6928964f6364.slice/crio-c877f544b779b9df42e3598f61726db850464701b905f26244bce314c9f85259 WatchSource:0}: Error finding container c877f544b779b9df42e3598f61726db850464701b905f26244bce314c9f85259: Status 404 returned error can't find the container with id c877f544b779b9df42e3598f61726db850464701b905f26244bce314c9f85259
Oct 30 21:38:19.032941 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:19.032906    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c746f769b-7h4sb" event={"ID":"b09d9c97-c4b6-4fbf-9652-6928964f6364","Type":"ContainerStarted","Data":"c877f544b779b9df42e3598f61726db850464701b905f26244bce314c9f85259"}
Oct 30 21:38:22.048003 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:22.047967    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c746f769b-7h4sb" event={"ID":"b09d9c97-c4b6-4fbf-9652-6928964f6364","Type":"ContainerStarted","Data":"9c37fef054a5694c564094ec0fe7940e35266a13a4a12ad83c7608ef1a43518e"}
Oct 30 21:38:22.703646 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:22.703572    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-mplpk"]
Oct 30 21:38:22.706793 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:22.706777    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-mplpk"
Oct 30 21:38:22.773102 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:22.773076    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 21:38:22.773329 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:22.773304    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-pjvfq\""
Oct 30 21:38:22.773473 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:22.773334    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 21:38:22.774585 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:22.774564    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-mplpk"]
Oct 30 21:38:22.821656 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:22.821628    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm55j\" (UniqueName: \"kubernetes.io/projected/f6eeaec5-a9fd-48cc-bfb2-10e0699d4c2f-kube-api-access-fm55j\") pod \"authorino-operator-7b7bb4bc77-mplpk\" (UID: \"f6eeaec5-a9fd-48cc-bfb2-10e0699d4c2f\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-mplpk"
Oct 30 21:38:22.922661 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:22.922633    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm55j\" (UniqueName: \"kubernetes.io/projected/f6eeaec5-a9fd-48cc-bfb2-10e0699d4c2f-kube-api-access-fm55j\") pod \"authorino-operator-7b7bb4bc77-mplpk\" (UID: \"f6eeaec5-a9fd-48cc-bfb2-10e0699d4c2f\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-mplpk"
Oct 30 21:38:23.002521 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:23.002496    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm55j\" (UniqueName: \"kubernetes.io/projected/f6eeaec5-a9fd-48cc-bfb2-10e0699d4c2f-kube-api-access-fm55j\") pod \"authorino-operator-7b7bb4bc77-mplpk\" (UID: \"f6eeaec5-a9fd-48cc-bfb2-10e0699d4c2f\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-mplpk"
Oct 30 21:38:23.019289 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:23.019266    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-mplpk"
Oct 30 21:38:23.055257 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:23.055227    5087 generic.go:356] "Generic (PLEG): container finished" podID="b09d9c97-c4b6-4fbf-9652-6928964f6364" containerID="9c37fef054a5694c564094ec0fe7940e35266a13a4a12ad83c7608ef1a43518e" exitCode=0
Oct 30 21:38:23.055741 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:23.055288    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c746f769b-7h4sb" event={"ID":"b09d9c97-c4b6-4fbf-9652-6928964f6364","Type":"ContainerDied","Data":"9c37fef054a5694c564094ec0fe7940e35266a13a4a12ad83c7608ef1a43518e"}
Oct 30 21:38:23.407310 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:23.407235    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-mplpk"]
Oct 30 21:38:23.410760 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:38:23.410724    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6eeaec5_a9fd_48cc_bfb2_10e0699d4c2f.slice/crio-d3d5d280bd00b449a09f7bdf9665e7c3f44607bf316cf6be116c4d8d9cb5b8eb WatchSource:0}: Error finding container d3d5d280bd00b449a09f7bdf9665e7c3f44607bf316cf6be116c4d8d9cb5b8eb: Status 404 returned error can't find the container with id d3d5d280bd00b449a09f7bdf9665e7c3f44607bf316cf6be116c4d8d9cb5b8eb
Oct 30 21:38:24.060438 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:24.060404    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c746f769b-7h4sb" event={"ID":"b09d9c97-c4b6-4fbf-9652-6928964f6364","Type":"ContainerStarted","Data":"f22272f9edf8283c0055cf00fccd9e770f7cb0e381d4032f381f65bf001d6d8e"}
Oct 30 21:38:24.060885 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:24.060536    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c746f769b-7h4sb"
Oct 30 21:38:24.066801 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:24.066499    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-mplpk" event={"ID":"f6eeaec5-a9fd-48cc-bfb2-10e0699d4c2f","Type":"ContainerStarted","Data":"d3d5d280bd00b449a09f7bdf9665e7c3f44607bf316cf6be116c4d8d9cb5b8eb"}
Oct 30 21:38:24.227648 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:24.227551    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c746f769b-7h4sb" podStartSLOduration=2.62138769 podStartE2EDuration="7.227538591s" totalImagesPullingTime="4.149547564s" totalInitContainerRuntime="456.603337ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:24.22669798 +0000 UTC m=+298.701201961" watchObservedRunningTime="2026-10-30 21:38:24.227538591 +0000 UTC m=+298.702042561"
Oct 30 21:38:25.737918 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:25.737888    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 21:38:25.738427 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:25.738377    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 21:38:25.746257 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:25.746236    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 21:38:25.746794 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:25.746773    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 21:38:27.288868 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:27.288834    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-jdkxk"]
Oct 30 21:38:27.295265 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:27.295246    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-jdkxk"
Oct 30 21:38:27.361075 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:27.361044    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hkhz\" (UniqueName: \"kubernetes.io/projected/c599a977-d2f9-4589-b4df-8590b31744d3-kube-api-access-6hkhz\") pod \"limitador-operator-controller-manager-5778bb4449-jdkxk\" (UID: \"c599a977-d2f9-4589-b4df-8590b31744d3\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-jdkxk"
Oct 30 21:38:27.390612 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:27.390581    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-zbd9t\""
Oct 30 21:38:27.391649 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:27.391629    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-jdkxk"]
Oct 30 21:38:27.461932 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:27.461905    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hkhz\" (UniqueName: \"kubernetes.io/projected/c599a977-d2f9-4589-b4df-8590b31744d3-kube-api-access-6hkhz\") pod \"limitador-operator-controller-manager-5778bb4449-jdkxk\" (UID: \"c599a977-d2f9-4589-b4df-8590b31744d3\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-jdkxk"
Oct 30 21:38:27.528711 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:27.528679    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hkhz\" (UniqueName: \"kubernetes.io/projected/c599a977-d2f9-4589-b4df-8590b31744d3-kube-api-access-6hkhz\") pod \"limitador-operator-controller-manager-5778bb4449-jdkxk\" (UID: \"c599a977-d2f9-4589-b4df-8590b31744d3\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-jdkxk"
Oct 30 21:38:27.608221 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:27.608155    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-jdkxk"
Oct 30 21:38:27.978583 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:27.978506    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-jdkxk"]
Oct 30 21:38:28.010547 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:38:28.010520    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc599a977_d2f9_4589_b4df_8590b31744d3.slice/crio-8f606ef9560d5cd8c5591c7112596bfe39b701e70e58fcabadc336370c8cff59 WatchSource:0}: Error finding container 8f606ef9560d5cd8c5591c7112596bfe39b701e70e58fcabadc336370c8cff59: Status 404 returned error can't find the container with id 8f606ef9560d5cd8c5591c7112596bfe39b701e70e58fcabadc336370c8cff59
Oct 30 21:38:28.012883 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:28.012862    5087 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:38:28.084067 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:28.084036    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-jdkxk" event={"ID":"c599a977-d2f9-4589-b4df-8590b31744d3","Type":"ContainerStarted","Data":"8f606ef9560d5cd8c5591c7112596bfe39b701e70e58fcabadc336370c8cff59"}
Oct 30 21:38:32.106735 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:32.106691    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-mplpk" event={"ID":"f6eeaec5-a9fd-48cc-bfb2-10e0699d4c2f","Type":"ContainerStarted","Data":"660ee45aff4982da1cf5f757a97153f7ca50f44cc6934de453d393b52beb87bc"}
Oct 30 21:38:32.107174 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:32.106758    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-mplpk"
Oct 30 21:38:32.108467 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:32.108441    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-jdkxk" event={"ID":"c599a977-d2f9-4589-b4df-8590b31744d3","Type":"ContainerStarted","Data":"680f625691fee117c0ff13c676f9e72ef61f502c7b8f74ef7da6c14766e2a3cd"}
Oct 30 21:38:32.108581 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:32.108502    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-jdkxk"
Oct 30 21:38:32.282036 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:32.281985    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-mplpk" podStartSLOduration=2.671718826 podStartE2EDuration="10.281968901s" totalImagesPullingTime="7.610250075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:32.280659435 +0000 UTC m=+306.755163418" watchObservedRunningTime="2026-10-30 21:38:32.281968901 +0000 UTC m=+306.756472883"
Oct 30 21:38:32.399564 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:32.399458    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-jdkxk" podStartSLOduration=2.282799696 podStartE2EDuration="5.399445382s" totalImagesPullingTime="3.116645686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:32.397281755 +0000 UTC m=+306.871785733" watchObservedRunningTime="2026-10-30 21:38:32.399445382 +0000 UTC m=+306.873949359"
Oct 30 21:38:34.469563 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:34.469490    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x4nzx"]
Oct 30 21:38:34.472880 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:34.472865    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x4nzx"
Oct 30 21:38:34.526019 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:34.525993    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lsrdr\""
Oct 30 21:38:34.576816 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:34.576788    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x4nzx"]
Oct 30 21:38:34.624537 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:34.624513    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt7zb\" (UniqueName: \"kubernetes.io/projected/385ea17a-3961-4253-b1b1-e8beaf700048-kube-api-access-kt7zb\") pod \"kuadrant-operator-controller-manager-678d87766f-x4nzx\" (UID: \"385ea17a-3961-4253-b1b1-e8beaf700048\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x4nzx"
Oct 30 21:38:34.624692 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:34.624541    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/385ea17a-3961-4253-b1b1-e8beaf700048-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-x4nzx\" (UID: \"385ea17a-3961-4253-b1b1-e8beaf700048\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x4nzx"
Oct 30 21:38:34.725969 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:34.725887    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt7zb\" (UniqueName: \"kubernetes.io/projected/385ea17a-3961-4253-b1b1-e8beaf700048-kube-api-access-kt7zb\") pod \"kuadrant-operator-controller-manager-678d87766f-x4nzx\" (UID: \"385ea17a-3961-4253-b1b1-e8beaf700048\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x4nzx"
Oct 30 21:38:34.725969 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:34.725928    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/385ea17a-3961-4253-b1b1-e8beaf700048-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-x4nzx\" (UID: \"385ea17a-3961-4253-b1b1-e8beaf700048\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x4nzx"
Oct 30 21:38:34.726408 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:34.726390    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/385ea17a-3961-4253-b1b1-e8beaf700048-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-x4nzx\" (UID: \"385ea17a-3961-4253-b1b1-e8beaf700048\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x4nzx"
Oct 30 21:38:34.783522 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:34.783487    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt7zb\" (UniqueName: \"kubernetes.io/projected/385ea17a-3961-4253-b1b1-e8beaf700048-kube-api-access-kt7zb\") pod \"kuadrant-operator-controller-manager-678d87766f-x4nzx\" (UID: \"385ea17a-3961-4253-b1b1-e8beaf700048\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x4nzx"
Oct 30 21:38:34.784257 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:34.784226    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x4nzx"
Oct 30 21:38:35.073460 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:35.073428    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7c746f769b-7h4sb"
Oct 30 21:38:35.279272 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:35.279188    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x4nzx"]
Oct 30 21:38:35.282229 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:38:35.282187    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod385ea17a_3961_4253_b1b1_e8beaf700048.slice/crio-4430758f7674a218b6c0b7e5d21b99686e0e2d48c03f1bf0dbc63ce14c126d56 WatchSource:0}: Error finding container 4430758f7674a218b6c0b7e5d21b99686e0e2d48c03f1bf0dbc63ce14c126d56: Status 404 returned error can't find the container with id 4430758f7674a218b6c0b7e5d21b99686e0e2d48c03f1bf0dbc63ce14c126d56
Oct 30 21:38:36.125006 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:36.124968    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x4nzx" event={"ID":"385ea17a-3961-4253-b1b1-e8beaf700048","Type":"ContainerStarted","Data":"4430758f7674a218b6c0b7e5d21b99686e0e2d48c03f1bf0dbc63ce14c126d56"}
Oct 30 21:38:42.157264 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:42.157231    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x4nzx" event={"ID":"385ea17a-3961-4253-b1b1-e8beaf700048","Type":"ContainerStarted","Data":"aca6bfc01649a5dd9d7cce880d747f697e6f2cda7f5a8a46416af9c1cf2d3699"}
Oct 30 21:38:42.157648 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:42.157378    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x4nzx"
Oct 30 21:38:42.372871 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:42.372817    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x4nzx" podStartSLOduration=2.395091567 podStartE2EDuration="8.37280335s" totalImagesPullingTime="5.977711783s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:42.371579971 +0000 UTC m=+316.846083970" watchObservedRunningTime="2026-10-30 21:38:42.37280335 +0000 UTC m=+316.847307329"
Oct 30 21:38:43.114406 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:43.114376    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-jdkxk"
Oct 30 21:38:43.114570 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:43.114446    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-mplpk"
Oct 30 21:38:47.167364 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:38:47.167314    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:38:53.163939 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:38:53.163908    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x4nzx"
Oct 30 21:39:17.167815 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:39:17.167776    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:39:20.402573 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.402543    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"]
Oct 30 21:39:20.406503 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.406486    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:20.500913 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.500872    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b05d7b91-0d6e-460e-b5c3-40a261c91e1f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf\" (UID: \"b05d7b91-0d6e-460e-b5c3-40a261c91e1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:20.501071 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.500989    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b05d7b91-0d6e-460e-b5c3-40a261c91e1f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf\" (UID: \"b05d7b91-0d6e-460e-b5c3-40a261c91e1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:20.501071 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.501031    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b05d7b91-0d6e-460e-b5c3-40a261c91e1f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf\" (UID: \"b05d7b91-0d6e-460e-b5c3-40a261c91e1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:20.501071 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.501050    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b05d7b91-0d6e-460e-b5c3-40a261c91e1f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf\" (UID: \"b05d7b91-0d6e-460e-b5c3-40a261c91e1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:20.501071 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.501067    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b05d7b91-0d6e-460e-b5c3-40a261c91e1f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf\" (UID: \"b05d7b91-0d6e-460e-b5c3-40a261c91e1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:20.501251 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.501092    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b05d7b91-0d6e-460e-b5c3-40a261c91e1f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf\" (UID: \"b05d7b91-0d6e-460e-b5c3-40a261c91e1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:20.501251 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.501132    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs2mg\" (UniqueName: \"kubernetes.io/projected/b05d7b91-0d6e-460e-b5c3-40a261c91e1f-kube-api-access-cs2mg\") pod \"data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf\" (UID: \"b05d7b91-0d6e-460e-b5c3-40a261c91e1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:20.501328 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.501303    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b05d7b91-0d6e-460e-b5c3-40a261c91e1f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf\" (UID: \"b05d7b91-0d6e-460e-b5c3-40a261c91e1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:20.501374 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.501357    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b05d7b91-0d6e-460e-b5c3-40a261c91e1f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf\" (UID: \"b05d7b91-0d6e-460e-b5c3-40a261c91e1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:20.513040 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.513015    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-82mvq\""
Oct 30 21:39:20.514523 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.514500    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"]
Oct 30 21:39:20.602532 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.602498    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b05d7b91-0d6e-460e-b5c3-40a261c91e1f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf\" (UID: \"b05d7b91-0d6e-460e-b5c3-40a261c91e1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:20.602732 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.602545    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b05d7b91-0d6e-460e-b5c3-40a261c91e1f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf\" (UID: \"b05d7b91-0d6e-460e-b5c3-40a261c91e1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:20.602732 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.602574    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b05d7b91-0d6e-460e-b5c3-40a261c91e1f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf\" (UID: \"b05d7b91-0d6e-460e-b5c3-40a261c91e1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:20.602732 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.602685    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b05d7b91-0d6e-460e-b5c3-40a261c91e1f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf\" (UID: \"b05d7b91-0d6e-460e-b5c3-40a261c91e1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:20.602732 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.602710    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b05d7b91-0d6e-460e-b5c3-40a261c91e1f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf\" (UID: \"b05d7b91-0d6e-460e-b5c3-40a261c91e1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:20.602956 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.602736    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs2mg\" (UniqueName: \"kubernetes.io/projected/b05d7b91-0d6e-460e-b5c3-40a261c91e1f-kube-api-access-cs2mg\") pod \"data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf\" (UID: \"b05d7b91-0d6e-460e-b5c3-40a261c91e1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:20.602956 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.602783    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b05d7b91-0d6e-460e-b5c3-40a261c91e1f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf\" (UID: \"b05d7b91-0d6e-460e-b5c3-40a261c91e1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:20.602956 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.602807    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b05d7b91-0d6e-460e-b5c3-40a261c91e1f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf\" (UID: \"b05d7b91-0d6e-460e-b5c3-40a261c91e1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:20.602956 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.602850    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b05d7b91-0d6e-460e-b5c3-40a261c91e1f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf\" (UID: \"b05d7b91-0d6e-460e-b5c3-40a261c91e1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:20.603149 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.603053    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b05d7b91-0d6e-460e-b5c3-40a261c91e1f-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf\" (UID: \"b05d7b91-0d6e-460e-b5c3-40a261c91e1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:20.603229 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.603205    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b05d7b91-0d6e-460e-b5c3-40a261c91e1f-istio-data\") pod \"data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf\" (UID: \"b05d7b91-0d6e-460e-b5c3-40a261c91e1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:20.603288 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.603252    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b05d7b91-0d6e-460e-b5c3-40a261c91e1f-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf\" (UID: \"b05d7b91-0d6e-460e-b5c3-40a261c91e1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:20.603411 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.603379    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b05d7b91-0d6e-460e-b5c3-40a261c91e1f-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf\" (UID: \"b05d7b91-0d6e-460e-b5c3-40a261c91e1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:20.603475 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.603432    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b05d7b91-0d6e-460e-b5c3-40a261c91e1f-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf\" (UID: \"b05d7b91-0d6e-460e-b5c3-40a261c91e1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:20.605299 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.605280    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b05d7b91-0d6e-460e-b5c3-40a261c91e1f-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf\" (UID: \"b05d7b91-0d6e-460e-b5c3-40a261c91e1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:20.605585 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.605569    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b05d7b91-0d6e-460e-b5c3-40a261c91e1f-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf\" (UID: \"b05d7b91-0d6e-460e-b5c3-40a261c91e1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:20.660011 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.659949    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b05d7b91-0d6e-460e-b5c3-40a261c91e1f-istio-token\") pod \"data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf\" (UID: \"b05d7b91-0d6e-460e-b5c3-40a261c91e1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:20.660240 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.660214    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs2mg\" (UniqueName: \"kubernetes.io/projected/b05d7b91-0d6e-460e-b5c3-40a261c91e1f-kube-api-access-cs2mg\") pod \"data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf\" (UID: \"b05d7b91-0d6e-460e-b5c3-40a261c91e1f\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:20.717339 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:20.717312    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:21.150010 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:21.149986    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"]
Oct 30 21:39:21.152222 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:39:21.152196    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb05d7b91_0d6e_460e_b5c3_40a261c91e1f.slice/crio-12d8f0bb0d0a84dda220c46adffc5ca7100a4ca92173cc8883ae3adadf7aa072 WatchSource:0}: Error finding container 12d8f0bb0d0a84dda220c46adffc5ca7100a4ca92173cc8883ae3adadf7aa072: Status 404 returned error can't find the container with id 12d8f0bb0d0a84dda220c46adffc5ca7100a4ca92173cc8883ae3adadf7aa072
Oct 30 21:39:21.154251 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:21.154222    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:39:21.154312 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:21.154282    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:39:21.154376 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:21.154312    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:39:21.327418 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:21.327377    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf" event={"ID":"b05d7b91-0d6e-460e-b5c3-40a261c91e1f","Type":"ContainerStarted","Data":"82f68d637c8351ca72253fb92590f284cb5caf8bbcd30661b271f868ae134497"}
Oct 30 21:39:21.327617 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:21.327446    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf" event={"ID":"b05d7b91-0d6e-460e-b5c3-40a261c91e1f","Type":"ContainerStarted","Data":"12d8f0bb0d0a84dda220c46adffc5ca7100a4ca92173cc8883ae3adadf7aa072"}
Oct 30 21:39:21.444025 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:21.443927    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf" podStartSLOduration=1.443908448 podStartE2EDuration="1.443908448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:21.443071249 +0000 UTC m=+355.917575252" watchObservedRunningTime="2026-10-30 21:39:21.443908448 +0000 UTC m=+355.918412431"
Oct 30 21:39:21.717619 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:21.717555    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:22.722462 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:22.722428    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:23.134994 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:23.134966    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-kdrsc"]
Oct 30 21:39:23.137176 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:23.137161    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-kdrsc"
Oct 30 21:39:23.189605 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:23.189578    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5956v\""
Oct 30 21:39:23.192475 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:23.192423    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-kdrsc"]
Oct 30 21:39:23.225183 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:23.225158    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctz8t\" (UniqueName: \"kubernetes.io/projected/29acfac1-a00c-4841-9f27-2b5f1ac9f46a-kube-api-access-ctz8t\") pod \"authorino-57f7f8659f-kdrsc\" (UID: \"29acfac1-a00c-4841-9f27-2b5f1ac9f46a\") " pod="kuadrant-system/authorino-57f7f8659f-kdrsc"
Oct 30 21:39:23.326319 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:23.326284    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctz8t\" (UniqueName: \"kubernetes.io/projected/29acfac1-a00c-4841-9f27-2b5f1ac9f46a-kube-api-access-ctz8t\") pod \"authorino-57f7f8659f-kdrsc\" (UID: \"29acfac1-a00c-4841-9f27-2b5f1ac9f46a\") " pod="kuadrant-system/authorino-57f7f8659f-kdrsc"
Oct 30 21:39:23.336625 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:23.336592    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:23.343039 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:23.339721    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf"
Oct 30 21:39:23.385783 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:23.385721    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctz8t\" (UniqueName: \"kubernetes.io/projected/29acfac1-a00c-4841-9f27-2b5f1ac9f46a-kube-api-access-ctz8t\") pod \"authorino-57f7f8659f-kdrsc\" (UID: \"29acfac1-a00c-4841-9f27-2b5f1ac9f46a\") " pod="kuadrant-system/authorino-57f7f8659f-kdrsc"
Oct 30 21:39:23.460420 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:23.460392    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-kdrsc"
Oct 30 21:39:23.836167 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:23.836134    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-kdrsc"]
Oct 30 21:39:23.839071 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:39:23.839033    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29acfac1_a00c_4841_9f27_2b5f1ac9f46a.slice/crio-72410d9f60629cc7624019c81f28136be70560ea07015b0d11b871e1fdafc5c0 WatchSource:0}: Error finding container 72410d9f60629cc7624019c81f28136be70560ea07015b0d11b871e1fdafc5c0: Status 404 returned error can't find the container with id 72410d9f60629cc7624019c81f28136be70560ea07015b0d11b871e1fdafc5c0
Oct 30 21:39:24.341393 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:24.341335    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-kdrsc" event={"ID":"29acfac1-a00c-4841-9f27-2b5f1ac9f46a","Type":"ContainerStarted","Data":"72410d9f60629cc7624019c81f28136be70560ea07015b0d11b871e1fdafc5c0"}
Oct 30 21:39:26.079606 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:26.079575    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ct4cr"]
Oct 30 21:39:26.081825 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:26.081809    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-ct4cr"
Oct 30 21:39:26.153224 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:26.153192    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nztwt\" (UniqueName: \"kubernetes.io/projected/19dde19b-1939-433a-baf8-4c6497a189ec-kube-api-access-nztwt\") pod \"authorino-56c6c9ddd5-ct4cr\" (UID: \"19dde19b-1939-433a-baf8-4c6497a189ec\") " pod="kuadrant-system/authorino-56c6c9ddd5-ct4cr"
Oct 30 21:39:26.172666 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:26.172633    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ct4cr"]
Oct 30 21:39:26.254784 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:26.254756    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nztwt\" (UniqueName: \"kubernetes.io/projected/19dde19b-1939-433a-baf8-4c6497a189ec-kube-api-access-nztwt\") pod \"authorino-56c6c9ddd5-ct4cr\" (UID: \"19dde19b-1939-433a-baf8-4c6497a189ec\") " pod="kuadrant-system/authorino-56c6c9ddd5-ct4cr"
Oct 30 21:39:26.313301 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:26.313275    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nztwt\" (UniqueName: \"kubernetes.io/projected/19dde19b-1939-433a-baf8-4c6497a189ec-kube-api-access-nztwt\") pod \"authorino-56c6c9ddd5-ct4cr\" (UID: \"19dde19b-1939-433a-baf8-4c6497a189ec\") " pod="kuadrant-system/authorino-56c6c9ddd5-ct4cr"
Oct 30 21:39:26.393658 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:26.393575    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-ct4cr"
Oct 30 21:39:26.801936 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:26.801903    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ct4cr"]
Oct 30 21:39:26.804720 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:39:26.804694    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19dde19b_1939_433a_baf8_4c6497a189ec.slice/crio-00dadb142fee93a0bb63e525b3811896fd54c3a8dfc5f74a573abe32ee65c673 WatchSource:0}: Error finding container 00dadb142fee93a0bb63e525b3811896fd54c3a8dfc5f74a573abe32ee65c673: Status 404 returned error can't find the container with id 00dadb142fee93a0bb63e525b3811896fd54c3a8dfc5f74a573abe32ee65c673
Oct 30 21:39:27.356494 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:27.356455    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-ct4cr" event={"ID":"19dde19b-1939-433a-baf8-4c6497a189ec","Type":"ContainerStarted","Data":"00dadb142fee93a0bb63e525b3811896fd54c3a8dfc5f74a573abe32ee65c673"}
Oct 30 21:39:27.357926 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:27.357896    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-kdrsc" event={"ID":"29acfac1-a00c-4841-9f27-2b5f1ac9f46a","Type":"ContainerStarted","Data":"08420dd690c24e244acc08328b13fcdd1932118a52e38a7c0f54f05319c4bdb6"}
Oct 30 21:39:27.472233 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:27.472186    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-kdrsc" podStartSLOduration=1.790465313 podStartE2EDuration="4.472174189s" totalImagesPullingTime="2.681708876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:27.471850388 +0000 UTC m=+361.946354368" watchObservedRunningTime="2026-10-30 21:39:27.472174189 +0000 UTC m=+361.946678162"
Oct 30 21:39:28.370581 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:28.370543    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-ct4cr" event={"ID":"19dde19b-1939-433a-baf8-4c6497a189ec","Type":"ContainerStarted","Data":"97babee0b4a9d8321f86ad567252b574075ad8f9867157aae416405a964f7fa8"}
Oct 30 21:39:28.498766 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:28.498719    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-ct4cr" podStartSLOduration=1.17571083 podStartE2EDuration="2.498703473s" totalImagesPullingTime="1.322992643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:28.497458723 +0000 UTC m=+362.971962705" watchObservedRunningTime="2026-10-30 21:39:28.498703473 +0000 UTC m=+362.973207452"
Oct 30 21:39:28.550823 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:28.550791    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-kdrsc"]
Oct 30 21:39:29.375043 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:29.375005    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-kdrsc" podUID="29acfac1-a00c-4841-9f27-2b5f1ac9f46a" containerName="authorino" containerID="cri-o://08420dd690c24e244acc08328b13fcdd1932118a52e38a7c0f54f05319c4bdb6" gracePeriod=30
Oct 30 21:39:29.619918 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:29.619889    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-kdrsc"
Oct 30 21:39:29.683555 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:29.683503    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29acfac1-a00c-4841-9f27-2b5f1ac9f46a-kube-api-access-ctz8t\" (UniqueName: \"kubernetes.io/projected/29acfac1-a00c-4841-9f27-2b5f1ac9f46a-kube-api-access-ctz8t\") pod \"29acfac1-a00c-4841-9f27-2b5f1ac9f46a\" (UID: \"29acfac1-a00c-4841-9f27-2b5f1ac9f46a\") "
Oct 30 21:39:29.685655 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:29.685635    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29acfac1-a00c-4841-9f27-2b5f1ac9f46a-kube-api-access-ctz8t" pod "29acfac1-a00c-4841-9f27-2b5f1ac9f46a" (UID: "29acfac1-a00c-4841-9f27-2b5f1ac9f46a"). InnerVolumeSpecName "kube-api-access-ctz8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:29.784716 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:29.784680    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctz8t\" (UniqueName: \"kubernetes.io/projected/29acfac1-a00c-4841-9f27-2b5f1ac9f46a-kube-api-access-ctz8t\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:39:30.381767 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:30.381734    5087 generic.go:356] "Generic (PLEG): container finished" podID="29acfac1-a00c-4841-9f27-2b5f1ac9f46a" containerID="08420dd690c24e244acc08328b13fcdd1932118a52e38a7c0f54f05319c4bdb6" exitCode=0
Oct 30 21:39:30.382216 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:30.381782    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-kdrsc"
Oct 30 21:39:30.382216 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:30.381791    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-kdrsc" event={"ID":"29acfac1-a00c-4841-9f27-2b5f1ac9f46a","Type":"ContainerDied","Data":"08420dd690c24e244acc08328b13fcdd1932118a52e38a7c0f54f05319c4bdb6"}
Oct 30 21:39:30.382216 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:30.381823    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-kdrsc" event={"ID":"29acfac1-a00c-4841-9f27-2b5f1ac9f46a","Type":"ContainerDied","Data":"72410d9f60629cc7624019c81f28136be70560ea07015b0d11b871e1fdafc5c0"}
Oct 30 21:39:30.382216 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:30.381839    5087 scope.go:122] "RemoveContainer" containerID="08420dd690c24e244acc08328b13fcdd1932118a52e38a7c0f54f05319c4bdb6"
Oct 30 21:39:30.390277 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:30.390259    5087 scope.go:122] "RemoveContainer" containerID="08420dd690c24e244acc08328b13fcdd1932118a52e38a7c0f54f05319c4bdb6"
Oct 30 21:39:30.390572 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:39:30.390555    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08420dd690c24e244acc08328b13fcdd1932118a52e38a7c0f54f05319c4bdb6\": container with ID starting with 08420dd690c24e244acc08328b13fcdd1932118a52e38a7c0f54f05319c4bdb6 not found: ID does not exist" containerID="08420dd690c24e244acc08328b13fcdd1932118a52e38a7c0f54f05319c4bdb6"
Oct 30 21:39:30.390640 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:30.390583    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08420dd690c24e244acc08328b13fcdd1932118a52e38a7c0f54f05319c4bdb6"} err="failed to get container status \"08420dd690c24e244acc08328b13fcdd1932118a52e38a7c0f54f05319c4bdb6\": rpc error: code = NotFound desc = could not find container \"08420dd690c24e244acc08328b13fcdd1932118a52e38a7c0f54f05319c4bdb6\": container with ID starting with 08420dd690c24e244acc08328b13fcdd1932118a52e38a7c0f54f05319c4bdb6 not found: ID does not exist"
Oct 30 21:39:30.545820 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:30.545782    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-kdrsc"]
Oct 30 21:39:30.547167 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:30.547145    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-kdrsc"]
Oct 30 21:39:31.707017 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:31.706983    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29acfac1-a00c-4841-9f27-2b5f1ac9f46a" path="/var/lib/kubelet/pods/29acfac1-a00c-4841-9f27-2b5f1ac9f46a/volumes"
Oct 30 21:39:32.936112 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:32.936080    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-swnjl"]
Oct 30 21:39:32.936518 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:32.936442    5087 cpu_manager.go:414] "containerMap: removing container" podUID="29acfac1-a00c-4841-9f27-2b5f1ac9f46a" containerName="authorino"
Oct 30 21:39:32.936518 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:32.936452    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29acfac1-a00c-4841-9f27-2b5f1ac9f46a" containerName="authorino"
Oct 30 21:39:32.936585 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:32.936531    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="29acfac1-a00c-4841-9f27-2b5f1ac9f46a" containerName="authorino"
Oct 30 21:39:32.939207 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:32.939191    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-swnjl"
Oct 30 21:39:32.992629 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:32.992608    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-wbkg9\""
Oct 30 21:39:33.013496 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:33.013473    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/03fd0d5a-b3d4-40c3-932c-af7a51959c3b-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-swnjl\" (UID: \"03fd0d5a-b3d4-40c3-932c-af7a51959c3b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-swnjl"
Oct 30 21:39:33.013607 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:33.013524    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03fd0d5a-b3d4-40c3-932c-af7a51959c3b-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-swnjl\" (UID: \"03fd0d5a-b3d4-40c3-932c-af7a51959c3b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-swnjl"
Oct 30 21:39:33.013607 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:33.013588    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03fd0d5a-b3d4-40c3-932c-af7a51959c3b-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-swnjl\" (UID: \"03fd0d5a-b3d4-40c3-932c-af7a51959c3b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-swnjl"
Oct 30 21:39:33.013607 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:33.013604    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pvrt\" (UniqueName: \"kubernetes.io/projected/03fd0d5a-b3d4-40c3-932c-af7a51959c3b-kube-api-access-2pvrt\") pod \"kserve-module-controller-manager-7566cf5fc8-swnjl\" (UID: \"03fd0d5a-b3d4-40c3-932c-af7a51959c3b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-swnjl"
Oct 30 21:39:33.041051 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:33.041029    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 21:39:33.041051 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:33.041037    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 21:39:33.114740 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:33.114711    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03fd0d5a-b3d4-40c3-932c-af7a51959c3b-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-swnjl\" (UID: \"03fd0d5a-b3d4-40c3-932c-af7a51959c3b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-swnjl"
Oct 30 21:39:33.114740 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:33.114739    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pvrt\" (UniqueName: \"kubernetes.io/projected/03fd0d5a-b3d4-40c3-932c-af7a51959c3b-kube-api-access-2pvrt\") pod \"kserve-module-controller-manager-7566cf5fc8-swnjl\" (UID: \"03fd0d5a-b3d4-40c3-932c-af7a51959c3b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-swnjl"
Oct 30 21:39:33.114893 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:33.114790    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/03fd0d5a-b3d4-40c3-932c-af7a51959c3b-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-swnjl\" (UID: \"03fd0d5a-b3d4-40c3-932c-af7a51959c3b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-swnjl"
Oct 30 21:39:33.114893 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:33.114842    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03fd0d5a-b3d4-40c3-932c-af7a51959c3b-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-swnjl\" (UID: \"03fd0d5a-b3d4-40c3-932c-af7a51959c3b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-swnjl"
Oct 30 21:39:33.115104 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:33.115086    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03fd0d5a-b3d4-40c3-932c-af7a51959c3b-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-swnjl\" (UID: \"03fd0d5a-b3d4-40c3-932c-af7a51959c3b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-swnjl"
Oct 30 21:39:33.115162 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:33.115146    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/03fd0d5a-b3d4-40c3-932c-af7a51959c3b-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-swnjl\" (UID: \"03fd0d5a-b3d4-40c3-932c-af7a51959c3b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-swnjl"
Oct 30 21:39:33.140480 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:33.140450    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Oct 30 21:39:33.142337 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:33.142317    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-swnjl"]
Oct 30 21:39:33.145618 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:33.145595    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/03fd0d5a-b3d4-40c3-932c-af7a51959c3b-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-swnjl\" (UID: \"03fd0d5a-b3d4-40c3-932c-af7a51959c3b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-swnjl"
Oct 30 21:39:33.172268 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:33.172241    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pvrt\" (UniqueName: \"kubernetes.io/projected/03fd0d5a-b3d4-40c3-932c-af7a51959c3b-kube-api-access-2pvrt\") pod \"kserve-module-controller-manager-7566cf5fc8-swnjl\" (UID: \"03fd0d5a-b3d4-40c3-932c-af7a51959c3b\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-swnjl"
Oct 30 21:39:33.251167 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:33.251146    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-swnjl"
Oct 30 21:39:33.674138 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:33.674009    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-swnjl"]
Oct 30 21:39:33.676845 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:39:33.676810    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03fd0d5a_b3d4_40c3_932c_af7a51959c3b.slice/crio-7017aff006557f119eeadc6e03d04a13325388e4dc81825ff5e14755f3209e59 WatchSource:0}: Error finding container 7017aff006557f119eeadc6e03d04a13325388e4dc81825ff5e14755f3209e59: Status 404 returned error can't find the container with id 7017aff006557f119eeadc6e03d04a13325388e4dc81825ff5e14755f3209e59
Oct 30 21:39:34.401563 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:34.401531    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-swnjl" event={"ID":"03fd0d5a-b3d4-40c3-932c-af7a51959c3b","Type":"ContainerStarted","Data":"7017aff006557f119eeadc6e03d04a13325388e4dc81825ff5e14755f3209e59"}
Oct 30 21:39:36.413239 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:36.413193    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-swnjl" event={"ID":"03fd0d5a-b3d4-40c3-932c-af7a51959c3b","Type":"ContainerStarted","Data":"37ce66bd61bc556254072bd5d4896aa5cd293933f4a68df5eeae920bc0ae3637"}
Oct 30 21:39:36.413707 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:36.413265    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-swnjl"
Oct 30 21:39:36.544444 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:36.544393    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-swnjl" podStartSLOduration=1.907183522 podStartE2EDuration="4.544379844s" totalImagesPullingTime="2.637196322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:36.54378639 +0000 UTC m=+371.018290369" watchObservedRunningTime="2026-10-30 21:39:36.544379844 +0000 UTC m=+371.018883862"
Oct 30 21:39:45.992453 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:45.992415    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-jvnl5"]
Oct 30 21:39:45.994824 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:45.994808    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jvnl5"
Oct 30 21:39:46.046585 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:46.046560    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 30 21:39:46.095791 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:46.095760    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-52rhf\""
Oct 30 21:39:46.095932 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:46.095816    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:39:46.095932 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:46.095855    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:39:46.097366 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:46.097333    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-jvnl5"]
Oct 30 21:39:46.127736 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:46.127714    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4117714f-de1f-4a99-9338-9837664d51ba-data\") pod \"postgres-7cdbf6dc44-jvnl5\" (UID: \"4117714f-de1f-4a99-9338-9837664d51ba\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jvnl5"
Oct 30 21:39:46.127876 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:46.127767    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdchm\" (UniqueName: \"kubernetes.io/projected/4117714f-de1f-4a99-9338-9837664d51ba-kube-api-access-mdchm\") pod \"postgres-7cdbf6dc44-jvnl5\" (UID: \"4117714f-de1f-4a99-9338-9837664d51ba\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jvnl5"
Oct 30 21:39:46.229227 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:46.229195    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4117714f-de1f-4a99-9338-9837664d51ba-data\") pod \"postgres-7cdbf6dc44-jvnl5\" (UID: \"4117714f-de1f-4a99-9338-9837664d51ba\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jvnl5"
Oct 30 21:39:46.229433 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:46.229271    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdchm\" (UniqueName: \"kubernetes.io/projected/4117714f-de1f-4a99-9338-9837664d51ba-kube-api-access-mdchm\") pod \"postgres-7cdbf6dc44-jvnl5\" (UID: \"4117714f-de1f-4a99-9338-9837664d51ba\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jvnl5"
Oct 30 21:39:46.229607 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:46.229580    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4117714f-de1f-4a99-9338-9837664d51ba-data\") pod \"postgres-7cdbf6dc44-jvnl5\" (UID: \"4117714f-de1f-4a99-9338-9837664d51ba\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jvnl5"
Oct 30 21:39:46.288179 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:46.288120    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdchm\" (UniqueName: \"kubernetes.io/projected/4117714f-de1f-4a99-9338-9837664d51ba-kube-api-access-mdchm\") pod \"postgres-7cdbf6dc44-jvnl5\" (UID: \"4117714f-de1f-4a99-9338-9837664d51ba\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jvnl5"
Oct 30 21:39:46.305938 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:46.305917    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jvnl5"
Oct 30 21:39:46.676680 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:46.676658    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-jvnl5"]
Oct 30 21:39:46.678905 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:39:46.678883    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4117714f_de1f_4a99_9338_9837664d51ba.slice/crio-b1cd1a49fed946d4ac6fb41c2d5a5abc3b345500bc4cc7a4a8401c67c072dd9d WatchSource:0}: Error finding container b1cd1a49fed946d4ac6fb41c2d5a5abc3b345500bc4cc7a4a8401c67c072dd9d: Status 404 returned error can't find the container with id b1cd1a49fed946d4ac6fb41c2d5a5abc3b345500bc4cc7a4a8401c67c072dd9d
Oct 30 21:39:47.166645 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:39:47.166612    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:39:47.462314 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:47.462215    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jvnl5" event={"ID":"4117714f-de1f-4a99-9338-9837664d51ba","Type":"ContainerStarted","Data":"b1cd1a49fed946d4ac6fb41c2d5a5abc3b345500bc4cc7a4a8401c67c072dd9d"}
Oct 30 21:39:53.494690 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:53.494660    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jvnl5" event={"ID":"4117714f-de1f-4a99-9338-9837664d51ba","Type":"ContainerStarted","Data":"e12d9d5e6eb1cfccaa7485ea850e7215814a94a9e79985b5977850f835db718c"}
Oct 30 21:39:53.495116 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:53.494787    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jvnl5"
Oct 30 21:39:53.609195 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:53.609141    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jvnl5" podStartSLOduration=2.387399577 podStartE2EDuration="8.609126147s" totalImagesPullingTime="6.22172657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:53.60783706 +0000 UTC m=+388.082341039" watchObservedRunningTime="2026-10-30 21:39:53.609126147 +0000 UTC m=+388.083630127"
Oct 30 21:39:57.419047 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:57.419012    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-swnjl"
Oct 30 21:39:59.530686 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:39:59.530659    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-jvnl5"
Oct 30 21:40:17.170061 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:40:17.169982    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:40:39.575531 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:39.575498    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-mnddg"]
Oct 30 21:40:39.577844 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:39.577829    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-mnddg"
Oct 30 21:40:39.631507 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:39.631482    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 30 21:40:39.680993 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:39.680964    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-jcbfj\""
Oct 30 21:40:39.682741 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:39.682722    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-mnddg"]
Oct 30 21:40:39.684115 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:39.684097    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-d4cf5"]
Oct 30 21:40:39.686468 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:39.686454    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4cf5"
Oct 30 21:40:39.717889 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:39.717866    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrs5b\" (UniqueName: \"kubernetes.io/projected/4319fba2-04c7-4072-806e-04db5ba1eeca-kube-api-access-lrs5b\") pod \"kserve-controller-manager-776cc66d47-mnddg\" (UID: \"4319fba2-04c7-4072-806e-04db5ba1eeca\") " pod="opendatahub/kserve-controller-manager-776cc66d47-mnddg"
Oct 30 21:40:39.718011 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:39.717899    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4319fba2-04c7-4072-806e-04db5ba1eeca-cert\") pod \"kserve-controller-manager-776cc66d47-mnddg\" (UID: \"4319fba2-04c7-4072-806e-04db5ba1eeca\") " pod="opendatahub/kserve-controller-manager-776cc66d47-mnddg"
Oct 30 21:40:39.738534 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:39.738494    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 30 21:40:39.740835 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:39.740806    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-d4cf5"]
Oct 30 21:40:39.741007 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:39.740992    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-6rnst\""
Oct 30 21:40:39.818788 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:39.818760    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/358a7d5b-3350-4806-b3c7-63335bac2a9d-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-d4cf5\" (UID: \"358a7d5b-3350-4806-b3c7-63335bac2a9d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4cf5"
Oct 30 21:40:39.818918 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:39.818817    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrs5b\" (UniqueName: \"kubernetes.io/projected/4319fba2-04c7-4072-806e-04db5ba1eeca-kube-api-access-lrs5b\") pod \"kserve-controller-manager-776cc66d47-mnddg\" (UID: \"4319fba2-04c7-4072-806e-04db5ba1eeca\") " pod="opendatahub/kserve-controller-manager-776cc66d47-mnddg"
Oct 30 21:40:39.818918 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:39.818840    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4319fba2-04c7-4072-806e-04db5ba1eeca-cert\") pod \"kserve-controller-manager-776cc66d47-mnddg\" (UID: \"4319fba2-04c7-4072-806e-04db5ba1eeca\") " pod="opendatahub/kserve-controller-manager-776cc66d47-mnddg"
Oct 30 21:40:39.819046 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:39.818946    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2tbl\" (UniqueName: \"kubernetes.io/projected/358a7d5b-3350-4806-b3c7-63335bac2a9d-kube-api-access-x2tbl\") pod \"llmisvc-controller-manager-869d5dcc5d-d4cf5\" (UID: \"358a7d5b-3350-4806-b3c7-63335bac2a9d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4cf5"
Oct 30 21:40:39.821489 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:39.821470    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4319fba2-04c7-4072-806e-04db5ba1eeca-cert\") pod \"kserve-controller-manager-776cc66d47-mnddg\" (UID: \"4319fba2-04c7-4072-806e-04db5ba1eeca\") " pod="opendatahub/kserve-controller-manager-776cc66d47-mnddg"
Oct 30 21:40:39.879913 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:39.879781    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrs5b\" (UniqueName: \"kubernetes.io/projected/4319fba2-04c7-4072-806e-04db5ba1eeca-kube-api-access-lrs5b\") pod \"kserve-controller-manager-776cc66d47-mnddg\" (UID: \"4319fba2-04c7-4072-806e-04db5ba1eeca\") " pod="opendatahub/kserve-controller-manager-776cc66d47-mnddg"
Oct 30 21:40:39.890576 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:39.890552    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-mnddg"
Oct 30 21:40:39.920429 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:39.920384    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2tbl\" (UniqueName: \"kubernetes.io/projected/358a7d5b-3350-4806-b3c7-63335bac2a9d-kube-api-access-x2tbl\") pod \"llmisvc-controller-manager-869d5dcc5d-d4cf5\" (UID: \"358a7d5b-3350-4806-b3c7-63335bac2a9d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4cf5"
Oct 30 21:40:39.920593 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:39.920443    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/358a7d5b-3350-4806-b3c7-63335bac2a9d-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-d4cf5\" (UID: \"358a7d5b-3350-4806-b3c7-63335bac2a9d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4cf5"
Oct 30 21:40:39.923253 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:39.923226    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/358a7d5b-3350-4806-b3c7-63335bac2a9d-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-d4cf5\" (UID: \"358a7d5b-3350-4806-b3c7-63335bac2a9d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4cf5"
Oct 30 21:40:39.978271 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:39.978243    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2tbl\" (UniqueName: \"kubernetes.io/projected/358a7d5b-3350-4806-b3c7-63335bac2a9d-kube-api-access-x2tbl\") pod \"llmisvc-controller-manager-869d5dcc5d-d4cf5\" (UID: \"358a7d5b-3350-4806-b3c7-63335bac2a9d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4cf5"
Oct 30 21:40:39.997772 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:39.997747    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4cf5"
Oct 30 21:40:40.277912 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:40.277890    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-mnddg"]
Oct 30 21:40:40.279829 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:40:40.279802    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4319fba2_04c7_4072_806e_04db5ba1eeca.slice/crio-fcdcf35bad43772bbef7616f65c37b57a504b4bba03d465bb4b567619a4fa549 WatchSource:0}: Error finding container fcdcf35bad43772bbef7616f65c37b57a504b4bba03d465bb4b567619a4fa549: Status 404 returned error can't find the container with id fcdcf35bad43772bbef7616f65c37b57a504b4bba03d465bb4b567619a4fa549
Oct 30 21:40:40.389155 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:40.389132    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-d4cf5"]
Oct 30 21:40:40.391601 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:40:40.391575    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod358a7d5b_3350_4806_b3c7_63335bac2a9d.slice/crio-eb4de37ca2cc5f93ae0d2b16d41982f063f5d9ab408b19e9a16d3629833668b7 WatchSource:0}: Error finding container eb4de37ca2cc5f93ae0d2b16d41982f063f5d9ab408b19e9a16d3629833668b7: Status 404 returned error can't find the container with id eb4de37ca2cc5f93ae0d2b16d41982f063f5d9ab408b19e9a16d3629833668b7
Oct 30 21:40:40.719659 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:40.719585    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4cf5" event={"ID":"358a7d5b-3350-4806-b3c7-63335bac2a9d","Type":"ContainerStarted","Data":"eb4de37ca2cc5f93ae0d2b16d41982f063f5d9ab408b19e9a16d3629833668b7"}
Oct 30 21:40:40.720864 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:40.720838    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-mnddg" event={"ID":"4319fba2-04c7-4072-806e-04db5ba1eeca","Type":"ContainerStarted","Data":"fcdcf35bad43772bbef7616f65c37b57a504b4bba03d465bb4b567619a4fa549"}
Oct 30 21:40:42.733707 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:42.733668    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-mnddg" event={"ID":"4319fba2-04c7-4072-806e-04db5ba1eeca","Type":"ContainerStarted","Data":"6be68413a38f43a660322a4d67e3146d75614491d616b458a1875b911d431393"}
Oct 30 21:40:42.734159 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:42.733914    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-mnddg"
Oct 30 21:40:42.858862 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:42.858754    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-mnddg" podStartSLOduration=1.585916528 podStartE2EDuration="3.858733464s" totalImagesPullingTime="2.272816936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:42.857723503 +0000 UTC m=+437.332227482" watchObservedRunningTime="2026-10-30 21:40:42.858733464 +0000 UTC m=+437.333237447"
Oct 30 21:40:43.739659 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:43.739616    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4cf5" event={"ID":"358a7d5b-3350-4806-b3c7-63335bac2a9d","Type":"ContainerStarted","Data":"6caefc1da1e2cedbd4e6efdc492207b41925263d08aabbd5f2f474e2ee894511"}
Oct 30 21:40:43.854996 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:43.854943    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4cf5" podStartSLOduration=2.064254055 podStartE2EDuration="4.854929025s" totalImagesPullingTime="2.79067497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:43.85369121 +0000 UTC m=+438.328195191" watchObservedRunningTime="2026-10-30 21:40:43.854929025 +0000 UTC m=+438.329433003"
Oct 30 21:40:44.747181 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:40:44.747144    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4cf5"
Oct 30 21:40:47.166634 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:40:47.166598    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:41:04.414967 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:04.414930    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-b56p8"]
Oct 30 21:41:04.419414 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:04.419393    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-b56p8"
Oct 30 21:41:04.473519 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:04.473496    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-b56p8"]
Oct 30 21:41:04.530547 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:04.530464    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48gdc\" (UniqueName: \"kubernetes.io/projected/e93c9796-e95b-44b2-8cc8-ee49bca213ef-kube-api-access-48gdc\") pod \"authorino-6f6944bcdd-b56p8\" (UID: \"e93c9796-e95b-44b2-8cc8-ee49bca213ef\") " pod="kuadrant-system/authorino-6f6944bcdd-b56p8"
Oct 30 21:41:04.632039 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:04.632009    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48gdc\" (UniqueName: \"kubernetes.io/projected/e93c9796-e95b-44b2-8cc8-ee49bca213ef-kube-api-access-48gdc\") pod \"authorino-6f6944bcdd-b56p8\" (UID: \"e93c9796-e95b-44b2-8cc8-ee49bca213ef\") " pod="kuadrant-system/authorino-6f6944bcdd-b56p8"
Oct 30 21:41:04.689224 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:04.689156    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48gdc\" (UniqueName: \"kubernetes.io/projected/e93c9796-e95b-44b2-8cc8-ee49bca213ef-kube-api-access-48gdc\") pod \"authorino-6f6944bcdd-b56p8\" (UID: \"e93c9796-e95b-44b2-8cc8-ee49bca213ef\") " pod="kuadrant-system/authorino-6f6944bcdd-b56p8"
Oct 30 21:41:04.731424 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:04.731397    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-b56p8"
Oct 30 21:41:04.830178 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:04.830146    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-b56p8"]
Oct 30 21:41:04.854075 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:04.854039    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5fcdf484c-x52fz"]
Oct 30 21:41:04.861592 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:04.861561    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fcdf484c-x52fz"
Oct 30 21:41:04.914884 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:04.914853    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fcdf484c-x52fz"]
Oct 30 21:41:04.934704 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:04.934676    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd2zm\" (UniqueName: \"kubernetes.io/projected/40d3b4ab-dcce-4702-a7af-ed662e077c18-kube-api-access-cd2zm\") pod \"authorino-5fcdf484c-x52fz\" (UID: \"40d3b4ab-dcce-4702-a7af-ed662e077c18\") " pod="kuadrant-system/authorino-5fcdf484c-x52fz"
Oct 30 21:41:05.035238 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:05.035204    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd2zm\" (UniqueName: \"kubernetes.io/projected/40d3b4ab-dcce-4702-a7af-ed662e077c18-kube-api-access-cd2zm\") pod \"authorino-5fcdf484c-x52fz\" (UID: \"40d3b4ab-dcce-4702-a7af-ed662e077c18\") " pod="kuadrant-system/authorino-5fcdf484c-x52fz"
Oct 30 21:41:05.091786 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:05.091757    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd2zm\" (UniqueName: \"kubernetes.io/projected/40d3b4ab-dcce-4702-a7af-ed662e077c18-kube-api-access-cd2zm\") pod \"authorino-5fcdf484c-x52fz\" (UID: \"40d3b4ab-dcce-4702-a7af-ed662e077c18\") " pod="kuadrant-system/authorino-5fcdf484c-x52fz"
Oct 30 21:41:05.098103 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:05.098075    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-b56p8"]
Oct 30 21:41:05.101658 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:41:05.101632    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode93c9796_e95b_44b2_8cc8_ee49bca213ef.slice/crio-8a2cc78b644ea0c32500f84c596283ac7ad197e2179b43afff39ae18958a39d1 WatchSource:0}: Error finding container 8a2cc78b644ea0c32500f84c596283ac7ad197e2179b43afff39ae18958a39d1: Status 404 returned error can't find the container with id 8a2cc78b644ea0c32500f84c596283ac7ad197e2179b43afff39ae18958a39d1
Oct 30 21:41:05.175454 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:05.175430    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fcdf484c-x52fz"
Oct 30 21:41:05.205964 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:05.205931    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5fcdf484c-x52fz"]
Oct 30 21:41:05.234191 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:05.234167    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-654d54bddf-f5vbj"]
Oct 30 21:41:05.239384 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:05.239360    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-654d54bddf-f5vbj"
Oct 30 21:41:05.291214 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:05.291141    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 21:41:05.292922 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:05.292897    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-654d54bddf-f5vbj"]
Oct 30 21:41:05.337672 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:05.337639    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f-tls-cert\") pod \"authorino-654d54bddf-f5vbj\" (UID: \"62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f\") " pod="kuadrant-system/authorino-654d54bddf-f5vbj"
Oct 30 21:41:05.337672 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:05.337671    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5h4m\" (UniqueName: \"kubernetes.io/projected/62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f-kube-api-access-c5h4m\") pod \"authorino-654d54bddf-f5vbj\" (UID: \"62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f\") " pod="kuadrant-system/authorino-654d54bddf-f5vbj"
Oct 30 21:41:05.439248 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:05.439216    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f-tls-cert\") pod \"authorino-654d54bddf-f5vbj\" (UID: \"62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f\") " pod="kuadrant-system/authorino-654d54bddf-f5vbj"
Oct 30 21:41:05.439248 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:05.439252    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5h4m\" (UniqueName: \"kubernetes.io/projected/62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f-kube-api-access-c5h4m\") pod \"authorino-654d54bddf-f5vbj\" (UID: \"62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f\") " pod="kuadrant-system/authorino-654d54bddf-f5vbj"
Oct 30 21:41:05.441792 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:05.441771    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f-tls-cert\") pod \"authorino-654d54bddf-f5vbj\" (UID: \"62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f\") " pod="kuadrant-system/authorino-654d54bddf-f5vbj"
Oct 30 21:41:05.495297 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:05.495270    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5h4m\" (UniqueName: \"kubernetes.io/projected/62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f-kube-api-access-c5h4m\") pod \"authorino-654d54bddf-f5vbj\" (UID: \"62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f\") " pod="kuadrant-system/authorino-654d54bddf-f5vbj"
Oct 30 21:41:05.545270 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:05.545106    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5fcdf484c-x52fz"]
Oct 30 21:41:05.547897 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:41:05.547866    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40d3b4ab_dcce_4702_a7af_ed662e077c18.slice/crio-a74cd74663bc6f81d3ee2221e9a0a9e6054b64b22c70cd1bbebbc869f5308e4e WatchSource:0}: Error finding container a74cd74663bc6f81d3ee2221e9a0a9e6054b64b22c70cd1bbebbc869f5308e4e: Status 404 returned error can't find the container with id a74cd74663bc6f81d3ee2221e9a0a9e6054b64b22c70cd1bbebbc869f5308e4e
Oct 30 21:41:05.553089 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:05.553073    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-654d54bddf-f5vbj"
Oct 30 21:41:05.847208 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:05.847130    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fcdf484c-x52fz" event={"ID":"40d3b4ab-dcce-4702-a7af-ed662e077c18","Type":"ContainerStarted","Data":"a74cd74663bc6f81d3ee2221e9a0a9e6054b64b22c70cd1bbebbc869f5308e4e"}
Oct 30 21:41:05.848303 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:05.848283    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-b56p8" event={"ID":"e93c9796-e95b-44b2-8cc8-ee49bca213ef","Type":"ContainerStarted","Data":"8a2cc78b644ea0c32500f84c596283ac7ad197e2179b43afff39ae18958a39d1"}
Oct 30 21:41:05.921582 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:05.921380    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-654d54bddf-f5vbj"]
Oct 30 21:41:05.923954 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:41:05.923927    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62bf9abc_d1e1_4cb0_ab25_ec5ded6af20f.slice/crio-bd401ae773a5ca23e878330097ca4380f0c54c1d1d9656cef073b95b0599656a WatchSource:0}: Error finding container bd401ae773a5ca23e878330097ca4380f0c54c1d1d9656cef073b95b0599656a: Status 404 returned error can't find the container with id bd401ae773a5ca23e878330097ca4380f0c54c1d1d9656cef073b95b0599656a
Oct 30 21:41:06.855006 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:06.854961    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-b56p8" event={"ID":"e93c9796-e95b-44b2-8cc8-ee49bca213ef","Type":"ContainerStarted","Data":"7942a9a7188016281bd0ea1139996a8ef008b495f3c8b86665eb6a837aa25cbd"}
Oct 30 21:41:06.855006 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:06.854978    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-b56p8" podUID="e93c9796-e95b-44b2-8cc8-ee49bca213ef" containerName="authorino" containerID="cri-o://7942a9a7188016281bd0ea1139996a8ef008b495f3c8b86665eb6a837aa25cbd" gracePeriod=30
Oct 30 21:41:06.856488 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:06.856463    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-654d54bddf-f5vbj" event={"ID":"62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f","Type":"ContainerStarted","Data":"bd401ae773a5ca23e878330097ca4380f0c54c1d1d9656cef073b95b0599656a"}
Oct 30 21:41:06.968696 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:06.968650    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-b56p8" podStartSLOduration=1.432094789 podStartE2EDuration="2.968635646s" totalImagesPullingTime="1.536540857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:06.967488436 +0000 UTC m=+461.441992406" watchObservedRunningTime="2026-10-30 21:41:06.968635646 +0000 UTC m=+461.443139624"
Oct 30 21:41:07.123026 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:07.123002    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-b56p8"
Oct 30 21:41:07.251287 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:07.251249    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e93c9796-e95b-44b2-8cc8-ee49bca213ef-kube-api-access-48gdc\" (UniqueName: \"kubernetes.io/projected/e93c9796-e95b-44b2-8cc8-ee49bca213ef-kube-api-access-48gdc\") pod \"e93c9796-e95b-44b2-8cc8-ee49bca213ef\" (UID: \"e93c9796-e95b-44b2-8cc8-ee49bca213ef\") "
Oct 30 21:41:07.254064 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:07.254034    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e93c9796-e95b-44b2-8cc8-ee49bca213ef-kube-api-access-48gdc" pod "e93c9796-e95b-44b2-8cc8-ee49bca213ef" (UID: "e93c9796-e95b-44b2-8cc8-ee49bca213ef"). InnerVolumeSpecName "kube-api-access-48gdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:41:07.352210 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:07.352182    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48gdc\" (UniqueName: \"kubernetes.io/projected/e93c9796-e95b-44b2-8cc8-ee49bca213ef-kube-api-access-48gdc\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:41:07.864779 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:07.864683    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fcdf484c-x52fz" event={"ID":"40d3b4ab-dcce-4702-a7af-ed662e077c18","Type":"ContainerStarted","Data":"d82c9f53e90e844ca62c0ab4e09fc35ce323f90b4dd1c423233e2d134f8e87a4"}
Oct 30 21:41:07.865260 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:07.864769    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5fcdf484c-x52fz" podUID="40d3b4ab-dcce-4702-a7af-ed662e077c18" containerName="authorino" containerID="cri-o://d82c9f53e90e844ca62c0ab4e09fc35ce323f90b4dd1c423233e2d134f8e87a4" gracePeriod=30
Oct 30 21:41:07.866657 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:07.866627    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-654d54bddf-f5vbj" event={"ID":"62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f","Type":"ContainerStarted","Data":"46288742bf10eff4495488d5c50bb8f68bed85740ee044141722d0b5a5c6a712"}
Oct 30 21:41:07.868084 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:07.868057    5087 generic.go:356] "Generic (PLEG): container finished" podID="e93c9796-e95b-44b2-8cc8-ee49bca213ef" containerID="7942a9a7188016281bd0ea1139996a8ef008b495f3c8b86665eb6a837aa25cbd" exitCode=0
Oct 30 21:41:07.868214 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:07.868162    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-b56p8" event={"ID":"e93c9796-e95b-44b2-8cc8-ee49bca213ef","Type":"ContainerDied","Data":"7942a9a7188016281bd0ea1139996a8ef008b495f3c8b86665eb6a837aa25cbd"}
Oct 30 21:41:07.868214 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:07.868193    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-b56p8" event={"ID":"e93c9796-e95b-44b2-8cc8-ee49bca213ef","Type":"ContainerDied","Data":"8a2cc78b644ea0c32500f84c596283ac7ad197e2179b43afff39ae18958a39d1"}
Oct 30 21:41:07.868338 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:07.868215    5087 scope.go:122] "RemoveContainer" containerID="7942a9a7188016281bd0ea1139996a8ef008b495f3c8b86665eb6a837aa25cbd"
Oct 30 21:41:07.868429 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:07.868372    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-b56p8"
Oct 30 21:41:07.878264 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:07.878246    5087 scope.go:122] "RemoveContainer" containerID="7942a9a7188016281bd0ea1139996a8ef008b495f3c8b86665eb6a837aa25cbd"
Oct 30 21:41:07.878638 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:41:07.878619    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7942a9a7188016281bd0ea1139996a8ef008b495f3c8b86665eb6a837aa25cbd\": container with ID starting with 7942a9a7188016281bd0ea1139996a8ef008b495f3c8b86665eb6a837aa25cbd not found: ID does not exist" containerID="7942a9a7188016281bd0ea1139996a8ef008b495f3c8b86665eb6a837aa25cbd"
Oct 30 21:41:07.878705 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:07.878645    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7942a9a7188016281bd0ea1139996a8ef008b495f3c8b86665eb6a837aa25cbd"} err="failed to get container status \"7942a9a7188016281bd0ea1139996a8ef008b495f3c8b86665eb6a837aa25cbd\": rpc error: code = NotFound desc = could not find container \"7942a9a7188016281bd0ea1139996a8ef008b495f3c8b86665eb6a837aa25cbd\": container with ID starting with 7942a9a7188016281bd0ea1139996a8ef008b495f3c8b86665eb6a837aa25cbd not found: ID does not exist"
Oct 30 21:41:07.978289 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:07.978230    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5fcdf484c-x52fz" podStartSLOduration=2.360690517 podStartE2EDuration="3.978210485s" totalImagesPullingTime="1.617519968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:07.976869858 +0000 UTC m=+462.451373837" watchObservedRunningTime="2026-10-30 21:41:07.978210485 +0000 UTC m=+462.452714464"
Oct 30 21:41:08.087547 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:08.087503    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-b56p8"]
Oct 30 21:41:08.089922 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:08.089895    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-b56p8"]
Oct 30 21:41:08.138002 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:08.137979    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fcdf484c-x52fz"
Oct 30 21:41:08.230008 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:08.229951    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-654d54bddf-f5vbj" podStartSLOduration=1.615653337 podStartE2EDuration="3.229930561s" totalImagesPullingTime="1.614277224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:08.202583352 +0000 UTC m=+462.677087332" watchObservedRunningTime="2026-10-30 21:41:08.229930561 +0000 UTC m=+462.704434540"
Oct 30 21:41:08.231032 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:08.231011    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ct4cr"]
Oct 30 21:41:08.231212 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:08.231194    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-ct4cr" podUID="19dde19b-1939-433a-baf8-4c6497a189ec" containerName="authorino" containerID="cri-o://97babee0b4a9d8321f86ad567252b574075ad8f9867157aae416405a964f7fa8" gracePeriod=30
Oct 30 21:41:08.263371 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:08.261060    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40d3b4ab-dcce-4702-a7af-ed662e077c18-kube-api-access-cd2zm\" (UniqueName: \"kubernetes.io/projected/40d3b4ab-dcce-4702-a7af-ed662e077c18-kube-api-access-cd2zm\") pod \"40d3b4ab-dcce-4702-a7af-ed662e077c18\" (UID: \"40d3b4ab-dcce-4702-a7af-ed662e077c18\") "
Oct 30 21:41:08.265036 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:08.265003    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d3b4ab-dcce-4702-a7af-ed662e077c18-kube-api-access-cd2zm" pod "40d3b4ab-dcce-4702-a7af-ed662e077c18" (UID: "40d3b4ab-dcce-4702-a7af-ed662e077c18"). InnerVolumeSpecName "kube-api-access-cd2zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:41:08.362566 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:08.362538    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cd2zm\" (UniqueName: \"kubernetes.io/projected/40d3b4ab-dcce-4702-a7af-ed662e077c18-kube-api-access-cd2zm\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:41:08.481809 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:08.481784    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-ct4cr"
Oct 30 21:41:08.564414 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:08.564383    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19dde19b-1939-433a-baf8-4c6497a189ec-kube-api-access-nztwt\" (UniqueName: \"kubernetes.io/projected/19dde19b-1939-433a-baf8-4c6497a189ec-kube-api-access-nztwt\") pod \"19dde19b-1939-433a-baf8-4c6497a189ec\" (UID: \"19dde19b-1939-433a-baf8-4c6497a189ec\") "
Oct 30 21:41:08.566573 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:08.566546    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19dde19b-1939-433a-baf8-4c6497a189ec-kube-api-access-nztwt" pod "19dde19b-1939-433a-baf8-4c6497a189ec" (UID: "19dde19b-1939-433a-baf8-4c6497a189ec"). InnerVolumeSpecName "kube-api-access-nztwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:41:08.665832 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:08.665764    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nztwt\" (UniqueName: \"kubernetes.io/projected/19dde19b-1939-433a-baf8-4c6497a189ec-kube-api-access-nztwt\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:41:08.874200 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:08.874166    5087 generic.go:356] "Generic (PLEG): container finished" podID="40d3b4ab-dcce-4702-a7af-ed662e077c18" containerID="d82c9f53e90e844ca62c0ab4e09fc35ce323f90b4dd1c423233e2d134f8e87a4" exitCode=0
Oct 30 21:41:08.874714 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:08.874210    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fcdf484c-x52fz"
Oct 30 21:41:08.874714 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:08.874241    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fcdf484c-x52fz" event={"ID":"40d3b4ab-dcce-4702-a7af-ed662e077c18","Type":"ContainerDied","Data":"d82c9f53e90e844ca62c0ab4e09fc35ce323f90b4dd1c423233e2d134f8e87a4"}
Oct 30 21:41:08.874714 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:08.874281    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fcdf484c-x52fz" event={"ID":"40d3b4ab-dcce-4702-a7af-ed662e077c18","Type":"ContainerDied","Data":"a74cd74663bc6f81d3ee2221e9a0a9e6054b64b22c70cd1bbebbc869f5308e4e"}
Oct 30 21:41:08.874714 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:08.874296    5087 scope.go:122] "RemoveContainer" containerID="d82c9f53e90e844ca62c0ab4e09fc35ce323f90b4dd1c423233e2d134f8e87a4"
Oct 30 21:41:08.875493 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:08.875467    5087 generic.go:356] "Generic (PLEG): container finished" podID="19dde19b-1939-433a-baf8-4c6497a189ec" containerID="97babee0b4a9d8321f86ad567252b574075ad8f9867157aae416405a964f7fa8" exitCode=0
Oct 30 21:41:08.875589 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:08.875512    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-ct4cr"
Oct 30 21:41:08.875589 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:08.875556    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-ct4cr" event={"ID":"19dde19b-1939-433a-baf8-4c6497a189ec","Type":"ContainerDied","Data":"97babee0b4a9d8321f86ad567252b574075ad8f9867157aae416405a964f7fa8"}
Oct 30 21:41:08.875697 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:08.875588    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-ct4cr" event={"ID":"19dde19b-1939-433a-baf8-4c6497a189ec","Type":"ContainerDied","Data":"00dadb142fee93a0bb63e525b3811896fd54c3a8dfc5f74a573abe32ee65c673"}
Oct 30 21:41:08.883662 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:08.883524    5087 scope.go:122] "RemoveContainer" containerID="d82c9f53e90e844ca62c0ab4e09fc35ce323f90b4dd1c423233e2d134f8e87a4"
Oct 30 21:41:08.883836 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:41:08.883810    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d82c9f53e90e844ca62c0ab4e09fc35ce323f90b4dd1c423233e2d134f8e87a4\": container with ID starting with d82c9f53e90e844ca62c0ab4e09fc35ce323f90b4dd1c423233e2d134f8e87a4 not found: ID does not exist" containerID="d82c9f53e90e844ca62c0ab4e09fc35ce323f90b4dd1c423233e2d134f8e87a4"
Oct 30 21:41:08.883911 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:08.883847    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d82c9f53e90e844ca62c0ab4e09fc35ce323f90b4dd1c423233e2d134f8e87a4"} err="failed to get container status \"d82c9f53e90e844ca62c0ab4e09fc35ce323f90b4dd1c423233e2d134f8e87a4\": rpc error: code = NotFound desc = could not find container \"d82c9f53e90e844ca62c0ab4e09fc35ce323f90b4dd1c423233e2d134f8e87a4\": container with ID starting with d82c9f53e90e844ca62c0ab4e09fc35ce323f90b4dd1c423233e2d134f8e87a4 not found: ID does not exist"
Oct 30 21:41:08.883911 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:08.883869    5087 scope.go:122] "RemoveContainer" containerID="97babee0b4a9d8321f86ad567252b574075ad8f9867157aae416405a964f7fa8"
Oct 30 21:41:08.891713 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:08.891697    5087 scope.go:122] "RemoveContainer" containerID="97babee0b4a9d8321f86ad567252b574075ad8f9867157aae416405a964f7fa8"
Oct 30 21:41:08.891942 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:41:08.891925    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97babee0b4a9d8321f86ad567252b574075ad8f9867157aae416405a964f7fa8\": container with ID starting with 97babee0b4a9d8321f86ad567252b574075ad8f9867157aae416405a964f7fa8 not found: ID does not exist" containerID="97babee0b4a9d8321f86ad567252b574075ad8f9867157aae416405a964f7fa8"
Oct 30 21:41:08.892009 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:08.891947    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97babee0b4a9d8321f86ad567252b574075ad8f9867157aae416405a964f7fa8"} err="failed to get container status \"97babee0b4a9d8321f86ad567252b574075ad8f9867157aae416405a964f7fa8\": rpc error: code = NotFound desc = could not find container \"97babee0b4a9d8321f86ad567252b574075ad8f9867157aae416405a964f7fa8\": container with ID starting with 97babee0b4a9d8321f86ad567252b574075ad8f9867157aae416405a964f7fa8 not found: ID does not exist"
Oct 30 21:41:09.051057 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:09.051015    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5fcdf484c-x52fz"]
Oct 30 21:41:09.058150 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:09.058115    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5fcdf484c-x52fz"]
Oct 30 21:41:09.174683 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:09.174652    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ct4cr"]
Oct 30 21:41:09.177237 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:09.177216    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-ct4cr"]
Oct 30 21:41:09.708570 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:09.708540    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19dde19b-1939-433a-baf8-4c6497a189ec" path="/var/lib/kubelet/pods/19dde19b-1939-433a-baf8-4c6497a189ec/volumes"
Oct 30 21:41:09.708934 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:09.708915    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40d3b4ab-dcce-4702-a7af-ed662e077c18" path="/var/lib/kubelet/pods/40d3b4ab-dcce-4702-a7af-ed662e077c18/volumes"
Oct 30 21:41:09.709279 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:09.709262    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e93c9796-e95b-44b2-8cc8-ee49bca213ef" path="/var/lib/kubelet/pods/e93c9796-e95b-44b2-8cc8-ee49bca213ef/volumes"
Oct 30 21:41:13.744802 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:13.744773    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-mnddg"
Oct 30 21:41:15.753641 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:15.753614    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4cf5"
Oct 30 21:41:17.166864 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:41:17.166826    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:41:47.166652 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:41:47.166576    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:41:49.496180 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:49.496145    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-s24zk"]
Oct 30 21:41:49.496612 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:49.496596    5087 cpu_manager.go:414] "containerMap: removing container" podUID="e93c9796-e95b-44b2-8cc8-ee49bca213ef" containerName="authorino"
Oct 30 21:41:49.496656 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:49.496614    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e93c9796-e95b-44b2-8cc8-ee49bca213ef" containerName="authorino"
Oct 30 21:41:49.496656 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:49.496636    5087 cpu_manager.go:414] "containerMap: removing container" podUID="19dde19b-1939-433a-baf8-4c6497a189ec" containerName="authorino"
Oct 30 21:41:49.496656 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:49.496641    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19dde19b-1939-433a-baf8-4c6497a189ec" containerName="authorino"
Oct 30 21:41:49.496748 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:49.496660    5087 cpu_manager.go:414] "containerMap: removing container" podUID="40d3b4ab-dcce-4702-a7af-ed662e077c18" containerName="authorino"
Oct 30 21:41:49.496748 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:49.496668    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40d3b4ab-dcce-4702-a7af-ed662e077c18" containerName="authorino"
Oct 30 21:41:49.496748 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:49.496739    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="19dde19b-1939-433a-baf8-4c6497a189ec" containerName="authorino"
Oct 30 21:41:49.496837 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:49.496751    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="40d3b4ab-dcce-4702-a7af-ed662e077c18" containerName="authorino"
Oct 30 21:41:49.496837 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:49.496759    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="e93c9796-e95b-44b2-8cc8-ee49bca213ef" containerName="authorino"
Oct 30 21:41:49.498945 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:49.498931    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s24zk"
Oct 30 21:41:49.527518 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:49.527490    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmfhm\" (UniqueName: \"kubernetes.io/projected/b85cc940-d5e6-4869-9763-881acf0ea4ac-kube-api-access-wmfhm\") pod \"llmisvc-controller-manager-6466c554d8-s24zk\" (UID: \"b85cc940-d5e6-4869-9763-881acf0ea4ac\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-s24zk"
Oct 30 21:41:49.527663 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:49.527552    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b85cc940-d5e6-4869-9763-881acf0ea4ac-cert\") pod \"llmisvc-controller-manager-6466c554d8-s24zk\" (UID: \"b85cc940-d5e6-4869-9763-881acf0ea4ac\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-s24zk"
Oct 30 21:41:49.556577 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:49.556553    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-s24zk"]
Oct 30 21:41:49.628338 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:49.628307    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmfhm\" (UniqueName: \"kubernetes.io/projected/b85cc940-d5e6-4869-9763-881acf0ea4ac-kube-api-access-wmfhm\") pod \"llmisvc-controller-manager-6466c554d8-s24zk\" (UID: \"b85cc940-d5e6-4869-9763-881acf0ea4ac\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-s24zk"
Oct 30 21:41:49.628505 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:49.628378    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b85cc940-d5e6-4869-9763-881acf0ea4ac-cert\") pod \"llmisvc-controller-manager-6466c554d8-s24zk\" (UID: \"b85cc940-d5e6-4869-9763-881acf0ea4ac\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-s24zk"
Oct 30 21:41:49.630874 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:49.630844    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b85cc940-d5e6-4869-9763-881acf0ea4ac-cert\") pod \"llmisvc-controller-manager-6466c554d8-s24zk\" (UID: \"b85cc940-d5e6-4869-9763-881acf0ea4ac\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-s24zk"
Oct 30 21:41:49.684335 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:49.684308    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmfhm\" (UniqueName: \"kubernetes.io/projected/b85cc940-d5e6-4869-9763-881acf0ea4ac-kube-api-access-wmfhm\") pod \"llmisvc-controller-manager-6466c554d8-s24zk\" (UID: \"b85cc940-d5e6-4869-9763-881acf0ea4ac\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-s24zk"
Oct 30 21:41:49.810948 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:49.810871    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s24zk"
Oct 30 21:41:50.177699 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:50.177658    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-s24zk"]
Oct 30 21:41:50.180749 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:41:50.180721    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85cc940_d5e6_4869_9763_881acf0ea4ac.slice/crio-85525ac07f2abd2e36f67dca9b5bee011e4fd753beab3e58058f85eee220bcc2 WatchSource:0}: Error finding container 85525ac07f2abd2e36f67dca9b5bee011e4fd753beab3e58058f85eee220bcc2: Status 404 returned error can't find the container with id 85525ac07f2abd2e36f67dca9b5bee011e4fd753beab3e58058f85eee220bcc2
Oct 30 21:41:51.075407 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:51.075370    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s24zk" event={"ID":"b85cc940-d5e6-4869-9763-881acf0ea4ac","Type":"ContainerStarted","Data":"fcdce8c2bd3e2b1beed3ad6ea3ffc3e901e2f89905b6135657bd46a0dff1669f"}
Oct 30 21:41:51.075893 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:51.075429    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s24zk" event={"ID":"b85cc940-d5e6-4869-9763-881acf0ea4ac","Type":"ContainerStarted","Data":"85525ac07f2abd2e36f67dca9b5bee011e4fd753beab3e58058f85eee220bcc2"}
Oct 30 21:41:51.075893 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:51.075469    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s24zk"
Oct 30 21:41:51.191090 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:41:51.190985    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s24zk" podStartSLOduration=1.437360701 podStartE2EDuration="2.190970606s" totalImagesPullingTime="753.609905ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:51.188987443 +0000 UTC m=+505.663491422" watchObservedRunningTime="2026-10-30 21:41:51.190970606 +0000 UTC m=+505.665474584"
Oct 30 21:42:17.166008 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:42:17.165969    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:42:22.082501 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:42:22.082466    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-s24zk"
Oct 30 21:42:22.247284 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:42:22.247251    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-d4cf5"]
Oct 30 21:42:22.247553 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:42:22.247530    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4cf5" podUID="358a7d5b-3350-4806-b3c7-63335bac2a9d" containerName="manager" containerID="cri-o://6caefc1da1e2cedbd4e6efdc492207b41925263d08aabbd5f2f474e2ee894511" gracePeriod=30
Oct 30 21:42:22.498238 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:42:22.498215    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4cf5"
Oct 30 21:42:22.522139 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:42:22.522117    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/358a7d5b-3350-4806-b3c7-63335bac2a9d-kube-api-access-x2tbl\" (UniqueName: \"kubernetes.io/projected/358a7d5b-3350-4806-b3c7-63335bac2a9d-kube-api-access-x2tbl\") pod \"358a7d5b-3350-4806-b3c7-63335bac2a9d\" (UID: \"358a7d5b-3350-4806-b3c7-63335bac2a9d\") "
Oct 30 21:42:22.522247 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:42:22.522159    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/358a7d5b-3350-4806-b3c7-63335bac2a9d-cert\" (UniqueName: \"kubernetes.io/secret/358a7d5b-3350-4806-b3c7-63335bac2a9d-cert\") pod \"358a7d5b-3350-4806-b3c7-63335bac2a9d\" (UID: \"358a7d5b-3350-4806-b3c7-63335bac2a9d\") "
Oct 30 21:42:22.524371 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:42:22.524333    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358a7d5b-3350-4806-b3c7-63335bac2a9d-kube-api-access-x2tbl" pod "358a7d5b-3350-4806-b3c7-63335bac2a9d" (UID: "358a7d5b-3350-4806-b3c7-63335bac2a9d"). InnerVolumeSpecName "kube-api-access-x2tbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:42:22.524471 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:42:22.524396    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/358a7d5b-3350-4806-b3c7-63335bac2a9d-cert" pod "358a7d5b-3350-4806-b3c7-63335bac2a9d" (UID: "358a7d5b-3350-4806-b3c7-63335bac2a9d"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:42:22.623130 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:42:22.623066    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2tbl\" (UniqueName: \"kubernetes.io/projected/358a7d5b-3350-4806-b3c7-63335bac2a9d-kube-api-access-x2tbl\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:42:22.623130 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:42:22.623090    5087 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/358a7d5b-3350-4806-b3c7-63335bac2a9d-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:42:23.222190 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:42:23.222153    5087 generic.go:356] "Generic (PLEG): container finished" podID="358a7d5b-3350-4806-b3c7-63335bac2a9d" containerID="6caefc1da1e2cedbd4e6efdc492207b41925263d08aabbd5f2f474e2ee894511" exitCode=0
Oct 30 21:42:23.222634 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:42:23.222193    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4cf5"
Oct 30 21:42:23.222634 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:42:23.222209    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4cf5" event={"ID":"358a7d5b-3350-4806-b3c7-63335bac2a9d","Type":"ContainerDied","Data":"6caefc1da1e2cedbd4e6efdc492207b41925263d08aabbd5f2f474e2ee894511"}
Oct 30 21:42:23.222634 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:42:23.222259    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-d4cf5" event={"ID":"358a7d5b-3350-4806-b3c7-63335bac2a9d","Type":"ContainerDied","Data":"eb4de37ca2cc5f93ae0d2b16d41982f063f5d9ab408b19e9a16d3629833668b7"}
Oct 30 21:42:23.222634 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:42:23.222285    5087 scope.go:122] "RemoveContainer" containerID="6caefc1da1e2cedbd4e6efdc492207b41925263d08aabbd5f2f474e2ee894511"
Oct 30 21:42:23.231531 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:42:23.231508    5087 scope.go:122] "RemoveContainer" containerID="6caefc1da1e2cedbd4e6efdc492207b41925263d08aabbd5f2f474e2ee894511"
Oct 30 21:42:23.231769 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:42:23.231753    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6caefc1da1e2cedbd4e6efdc492207b41925263d08aabbd5f2f474e2ee894511\": container with ID starting with 6caefc1da1e2cedbd4e6efdc492207b41925263d08aabbd5f2f474e2ee894511 not found: ID does not exist" containerID="6caefc1da1e2cedbd4e6efdc492207b41925263d08aabbd5f2f474e2ee894511"
Oct 30 21:42:23.231814 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:42:23.231779    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6caefc1da1e2cedbd4e6efdc492207b41925263d08aabbd5f2f474e2ee894511"} err="failed to get container status \"6caefc1da1e2cedbd4e6efdc492207b41925263d08aabbd5f2f474e2ee894511\": rpc error: code = NotFound desc = could not find container \"6caefc1da1e2cedbd4e6efdc492207b41925263d08aabbd5f2f474e2ee894511\": container with ID starting with 6caefc1da1e2cedbd4e6efdc492207b41925263d08aabbd5f2f474e2ee894511 not found: ID does not exist"
Oct 30 21:42:23.389922 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:42:23.389884    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-d4cf5"]
Oct 30 21:42:23.393681 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:42:23.393657    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-d4cf5"]
Oct 30 21:42:23.708224 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:42:23.708190    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="358a7d5b-3350-4806-b3c7-63335bac2a9d" path="/var/lib/kubelet/pods/358a7d5b-3350-4806-b3c7-63335bac2a9d/volumes"
Oct 30 21:42:47.166235 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:42:47.166194    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:43:10.785790 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.785722    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"]
Oct 30 21:43:10.787201 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.786095    5087 cpu_manager.go:414] "containerMap: removing container" podUID="358a7d5b-3350-4806-b3c7-63335bac2a9d" containerName="manager"
Oct 30 21:43:10.787201 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.786105    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="358a7d5b-3350-4806-b3c7-63335bac2a9d" containerName="manager"
Oct 30 21:43:10.787201 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.786185    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="358a7d5b-3350-4806-b3c7-63335bac2a9d" containerName="manager"
Oct 30 21:43:10.788043 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.788018    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:10.852397 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.852369    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-xn2np\""
Oct 30 21:43:10.874905 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.874874    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-8xwmx\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:10.875052 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.874915    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-8xwmx\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:10.875052 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.874936    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-8xwmx\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:10.875052 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.874951    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4928c741-8a88-4452-b2ce-f7ed6cff5304-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-8xwmx\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:10.875161 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.875088    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-8xwmx\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:10.875161 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.875134    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4928c741-8a88-4452-b2ce-f7ed6cff5304-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-8xwmx\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:10.875226 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.875190    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-8xwmx\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:10.875259 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.875238    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4928c741-8a88-4452-b2ce-f7ed6cff5304-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-8xwmx\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:10.875306 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.875290    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52wxm\" (UniqueName: \"kubernetes.io/projected/4928c741-8a88-4452-b2ce-f7ed6cff5304-kube-api-access-52wxm\") pod \"maas-default-gateway-openshift-default-74854d8ccf-8xwmx\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:10.913098 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.913068    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"]
Oct 30 21:43:10.976172 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.976133    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4928c741-8a88-4452-b2ce-f7ed6cff5304-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-8xwmx\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:10.976386 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.976197    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52wxm\" (UniqueName: \"kubernetes.io/projected/4928c741-8a88-4452-b2ce-f7ed6cff5304-kube-api-access-52wxm\") pod \"maas-default-gateway-openshift-default-74854d8ccf-8xwmx\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:10.976386 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.976229    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-8xwmx\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:10.976386 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.976277    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-8xwmx\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:10.976386 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.976316    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-8xwmx\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:10.976386 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.976340    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4928c741-8a88-4452-b2ce-f7ed6cff5304-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-8xwmx\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:10.976638 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.976429    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-8xwmx\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:10.976638 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.976489    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4928c741-8a88-4452-b2ce-f7ed6cff5304-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-8xwmx\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:10.976638 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.976530    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-8xwmx\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:10.977223 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.977192    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-8xwmx\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:10.977377 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.977229    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4928c741-8a88-4452-b2ce-f7ed6cff5304-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-8xwmx\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:10.977500 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.977483    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-8xwmx\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:10.977572 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.977549    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-8xwmx\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:10.977672 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.977655    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-8xwmx\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:10.979442 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.979423    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-8xwmx\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:10.979708 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:10.979691    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4928c741-8a88-4452-b2ce-f7ed6cff5304-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-8xwmx\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:11.043242 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:11.043172    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52wxm\" (UniqueName: \"kubernetes.io/projected/4928c741-8a88-4452-b2ce-f7ed6cff5304-kube-api-access-52wxm\") pod \"maas-default-gateway-openshift-default-74854d8ccf-8xwmx\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:11.043372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:11.043318    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4928c741-8a88-4452-b2ce-f7ed6cff5304-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-8xwmx\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:11.099491 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:11.099449    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:11.525750 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:11.525724    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"]
Oct 30 21:43:11.527931 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:43:11.527892    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4928c741_8a88_4452_b2ce_f7ed6cff5304.slice/crio-1b6a8f6972c7358a375d6bf3e09d9bb71b8a6b8cc11b72501078e2ddc1c188fb WatchSource:0}: Error finding container 1b6a8f6972c7358a375d6bf3e09d9bb71b8a6b8cc11b72501078e2ddc1c188fb: Status 404 returned error can't find the container with id 1b6a8f6972c7358a375d6bf3e09d9bb71b8a6b8cc11b72501078e2ddc1c188fb
Oct 30 21:43:11.529950 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:11.529919    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:43:11.530028 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:11.529991    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:43:11.530028 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:11.530020    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:43:12.472609 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:12.472569    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx" event={"ID":"4928c741-8a88-4452-b2ce-f7ed6cff5304","Type":"ContainerStarted","Data":"4200d982d1b2efc9118d95eb28ec489523a5877766f8063891f88c208ed8be1a"}
Oct 30 21:43:12.473122 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:12.472632    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx" event={"ID":"4928c741-8a88-4452-b2ce-f7ed6cff5304","Type":"ContainerStarted","Data":"1b6a8f6972c7358a375d6bf3e09d9bb71b8a6b8cc11b72501078e2ddc1c188fb"}
Oct 30 21:43:12.704563 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:12.704497    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx" podStartSLOduration=2.704478662 podStartE2EDuration="2.704478662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:12.654006324 +0000 UTC m=+587.128510368" watchObservedRunningTime="2026-10-30 21:43:12.704478662 +0000 UTC m=+587.178982645"
Oct 30 21:43:13.100682 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:13.100648    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:13.102328 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:13.102303    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-8xwmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 21:43:13.102419 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:13.102369    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx" podUID="4928c741-8a88-4452-b2ce-f7ed6cff5304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 21:43:14.101303 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:14.101270    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-8xwmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 21:43:14.101803 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:14.101561    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx" podUID="4928c741-8a88-4452-b2ce-f7ed6cff5304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 21:43:15.099820 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:15.099783    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-8xwmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 21:43:15.099988 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:15.099842    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx" podUID="4928c741-8a88-4452-b2ce-f7ed6cff5304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 21:43:16.100831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:16.100795    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-8xwmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 21:43:16.101305 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:16.100856    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx" podUID="4928c741-8a88-4452-b2ce-f7ed6cff5304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 21:43:17.100702 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:17.100666    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-8xwmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 21:43:17.100885 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:17.100731    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx" podUID="4928c741-8a88-4452-b2ce-f7ed6cff5304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 21:43:17.185063 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:43:17.185027    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:43:18.100651 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:18.100609    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-8xwmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Oct 30 21:43:18.100851 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:18.100674    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx" podUID="4928c741-8a88-4452-b2ce-f7ed6cff5304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Oct 30 21:43:20.101096 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:20.101055    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-8xwmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:43:20.101573 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:20.101129    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx" podUID="4928c741-8a88-4452-b2ce-f7ed6cff5304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:43:21.142544 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:21.142497    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-8xwmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:43:21.142977 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:21.142578    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx" podUID="4928c741-8a88-4452-b2ce-f7ed6cff5304" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:43:21.289102 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:21.289068    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:21.289573 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:21.289546    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:21.290480 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:21.290457    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:43:25.771167 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:25.771139    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 21:43:25.772095 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:25.772074    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 21:43:25.779381 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:25.779364    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 21:43:25.780363 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:25.780334    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 21:43:25.861995 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:25.861949    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"]
Oct 30 21:43:25.865717 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:25.865697    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:25.942409 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:25.942375    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-4k5dm\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:25.942558 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:25.942434    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4279c46-56a1-4788-b768-76b46d2988db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-4k5dm\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:25.942606 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:25.942571    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-4k5dm\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:25.942674 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:25.942657    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4279c46-56a1-4788-b768-76b46d2988db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-4k5dm\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:25.942711 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:25.942694    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-4k5dm\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:25.942744 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:25.942715    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phsgn\" (UniqueName: \"kubernetes.io/projected/a4279c46-56a1-4788-b768-76b46d2988db-kube-api-access-phsgn\") pod \"maas-default-gateway-openshift-default-74854d8ccf-4k5dm\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:25.942744 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:25.942740    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-4k5dm\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:25.942807 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:25.942781    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4279c46-56a1-4788-b768-76b46d2988db-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-4k5dm\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:25.942919 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:25.942896    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-4k5dm\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:25.976196 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:25.976159    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"]
Oct 30 21:43:26.044549 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:26.044468    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-4k5dm\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:26.044549 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:26.044513    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4279c46-56a1-4788-b768-76b46d2988db-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-4k5dm\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:26.044779 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:26.044572    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-4k5dm\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:26.044779 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:26.044641    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-4k5dm\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:26.044779 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:26.044707    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4279c46-56a1-4788-b768-76b46d2988db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-4k5dm\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:26.044779 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:26.044740    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-4k5dm\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:26.044989 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:26.044798    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4279c46-56a1-4788-b768-76b46d2988db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-4k5dm\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:26.044989 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:26.044845    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-4k5dm\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:26.044989 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:26.044876    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phsgn\" (UniqueName: \"kubernetes.io/projected/a4279c46-56a1-4788-b768-76b46d2988db-kube-api-access-phsgn\") pod \"maas-default-gateway-openshift-default-74854d8ccf-4k5dm\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:26.045144 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:26.045008    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-4k5dm\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:26.045203 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:26.045172    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-4k5dm\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:26.045528 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:26.045503    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-4k5dm\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:26.045637 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:26.045536    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-4k5dm\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:26.045820 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:26.045798    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4279c46-56a1-4788-b768-76b46d2988db-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-4k5dm\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:26.047224 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:26.047203    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-4k5dm\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:26.047814 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:26.047798    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4279c46-56a1-4788-b768-76b46d2988db-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-4k5dm\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:26.107730 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:26.107694    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4279c46-56a1-4788-b768-76b46d2988db-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-4k5dm\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:26.107863 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:26.107698    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phsgn\" (UniqueName: \"kubernetes.io/projected/a4279c46-56a1-4788-b768-76b46d2988db-kube-api-access-phsgn\") pod \"maas-default-gateway-openshift-default-74854d8ccf-4k5dm\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:26.178245 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:26.178215    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:26.667702 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:26.667668    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"]
Oct 30 21:43:26.671773 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:43:26.671741    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4279c46_56a1_4788_b768_76b46d2988db.slice/crio-24767bb4ed9b7d7dd640685d954156d36b265673914bf302a9f3c81f51a1c7aa WatchSource:0}: Error finding container 24767bb4ed9b7d7dd640685d954156d36b265673914bf302a9f3c81f51a1c7aa: Status 404 returned error can't find the container with id 24767bb4ed9b7d7dd640685d954156d36b265673914bf302a9f3c81f51a1c7aa
Oct 30 21:43:26.674166 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:26.674136    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:43:26.674240 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:26.674202    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:43:26.674240 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:26.674230    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:43:27.563373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:27.563125    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm" event={"ID":"a4279c46-56a1-4788-b768-76b46d2988db","Type":"ContainerStarted","Data":"b7357cc279ea07362f500ecb30445763139d014bf50092c6c5b6f12d850d19c1"}
Oct 30 21:43:27.563373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:27.563192    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm" event={"ID":"a4279c46-56a1-4788-b768-76b46d2988db","Type":"ContainerStarted","Data":"24767bb4ed9b7d7dd640685d954156d36b265673914bf302a9f3c81f51a1c7aa"}
Oct 30 21:43:27.692229 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:27.692174    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm" podStartSLOduration=2.692159676 podStartE2EDuration="2.692159676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:27.690047917 +0000 UTC m=+602.164551895" watchObservedRunningTime="2026-10-30 21:43:27.692159676 +0000 UTC m=+602.166663654"
Oct 30 21:43:28.178881 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:28.178834    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:28.180112 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:28.180087    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-4k5dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 21:43:28.180249 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:28.180137    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm" podUID="a4279c46-56a1-4788-b768-76b46d2988db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 21:43:29.179023 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:29.178984    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-4k5dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 21:43:29.179555 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:29.179049    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm" podUID="a4279c46-56a1-4788-b768-76b46d2988db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 21:43:30.178996 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:30.178961    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-4k5dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 21:43:30.179176 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:30.179026    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm" podUID="a4279c46-56a1-4788-b768-76b46d2988db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 21:43:31.179008 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:31.178976    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-4k5dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 21:43:31.179187 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:31.179028    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm" podUID="a4279c46-56a1-4788-b768-76b46d2988db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 21:43:32.179033 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:32.178996    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-4k5dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 21:43:32.179232 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:32.179070    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm" podUID="a4279c46-56a1-4788-b768-76b46d2988db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 21:43:33.179309 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:33.179268    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-4k5dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 30 21:43:33.179731 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:33.179370    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm" podUID="a4279c46-56a1-4788-b768-76b46d2988db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 30 21:43:34.812329 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:34.812299    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:34.812831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:34.812728    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:34.813525 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:34.813509    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:43:40.459516 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:40.459485    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-654d54bddf-f5vbj"]
Oct 30 21:43:40.459935 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:40.459672    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-654d54bddf-f5vbj" podUID="62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f" containerName="authorino" containerID="cri-o://46288742bf10eff4495488d5c50bb8f68bed85740ee044141722d0b5a5c6a712" gracePeriod=30
Oct 30 21:43:40.628495 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:40.626538    5087 generic.go:356] "Generic (PLEG): container finished" podID="62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f" containerID="46288742bf10eff4495488d5c50bb8f68bed85740ee044141722d0b5a5c6a712" exitCode=0
Oct 30 21:43:40.628495 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:40.626623    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-654d54bddf-f5vbj" event={"ID":"62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f","Type":"ContainerDied","Data":"46288742bf10eff4495488d5c50bb8f68bed85740ee044141722d0b5a5c6a712"}
Oct 30 21:43:40.711169 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:40.711111    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-654d54bddf-f5vbj"
Oct 30 21:43:40.785602 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:40.785572    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f-tls-cert\" (UniqueName: \"kubernetes.io/secret/62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f-tls-cert\") pod \"62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f\" (UID: \"62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f\") "
Oct 30 21:43:40.785736 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:40.785622    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f-kube-api-access-c5h4m\" (UniqueName: \"kubernetes.io/projected/62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f-kube-api-access-c5h4m\") pod \"62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f\" (UID: \"62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f\") "
Oct 30 21:43:40.787766 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:40.787734    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f-kube-api-access-c5h4m" pod "62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f" (UID: "62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f"). InnerVolumeSpecName "kube-api-access-c5h4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:43:40.796795 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:40.796767    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f-tls-cert" pod "62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f" (UID: "62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:43:40.886950 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:40.886912    5087 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f-tls-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:43:40.886950 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:40.886937    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5h4m\" (UniqueName: \"kubernetes.io/projected/62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f-kube-api-access-c5h4m\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:43:41.633705 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:41.633679    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-654d54bddf-f5vbj"
Oct 30 21:43:41.634169 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:41.633700    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-654d54bddf-f5vbj" event={"ID":"62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f","Type":"ContainerDied","Data":"bd401ae773a5ca23e878330097ca4380f0c54c1d1d9656cef073b95b0599656a"}
Oct 30 21:43:41.634169 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:41.633753    5087 scope.go:122] "RemoveContainer" containerID="46288742bf10eff4495488d5c50bb8f68bed85740ee044141722d0b5a5c6a712"
Oct 30 21:43:41.799700 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:41.799668    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-654d54bddf-f5vbj"]
Oct 30 21:43:41.848884 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:41.848856    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-654d54bddf-f5vbj"]
Oct 30 21:43:43.707799 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:43:43.707764    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f" path="/var/lib/kubelet/pods/62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f/volumes"
Oct 30 21:43:47.166733 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:43:47.166696    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:44:17.166118 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:44:17.166074    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:44:26.391472 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.391432    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"]
Oct 30 21:44:26.391919 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.391909    5087 cpu_manager.go:414] "containerMap: removing container" podUID="62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f" containerName="authorino"
Oct 30 21:44:26.391965 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.391923    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f" containerName="authorino"
Oct 30 21:44:26.392006 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.391986    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="62bf9abc-d1e1-4cb0-ab25-ec5ded6af20f" containerName="authorino"
Oct 30 21:44:26.396359 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.396325    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:26.492873 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.492842    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/964d70b0-09d8-46d7-98fa-d034aa975096-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-rz8s5\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:26.493043 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.492882    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-rz8s5\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:26.493043 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.492926    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-rz8s5\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:26.493120 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.493033    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-rz8s5\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:26.493120 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.493076    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/964d70b0-09d8-46d7-98fa-d034aa975096-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-rz8s5\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:26.493184 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.493163    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-rz8s5\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:26.493217 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.493184    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/964d70b0-09d8-46d7-98fa-d034aa975096-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-rz8s5\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:26.493249 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.493232    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27pct\" (UniqueName: \"kubernetes.io/projected/964d70b0-09d8-46d7-98fa-d034aa975096-kube-api-access-27pct\") pod \"maas-default-gateway-openshift-default-74854d8ccf-rz8s5\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:26.493281 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.493255    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-rz8s5\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:26.594115 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.594081    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-rz8s5\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:26.594307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.594125    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-rz8s5\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:26.594307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.594156    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/964d70b0-09d8-46d7-98fa-d034aa975096-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-rz8s5\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:26.594307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.594190    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-rz8s5\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:26.594307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.594212    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/964d70b0-09d8-46d7-98fa-d034aa975096-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-rz8s5\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:26.594307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.594244    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27pct\" (UniqueName: \"kubernetes.io/projected/964d70b0-09d8-46d7-98fa-d034aa975096-kube-api-access-27pct\") pod \"maas-default-gateway-openshift-default-74854d8ccf-rz8s5\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:26.594612 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.594398    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-rz8s5\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:26.594612 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.594460    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/964d70b0-09d8-46d7-98fa-d034aa975096-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-rz8s5\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:26.594612 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.594502    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-rz8s5\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:26.594612 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.594560    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-rz8s5\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:26.594849 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.594809    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-rz8s5\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:26.594958 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.594850    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-rz8s5\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:26.594958 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.594913    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-rz8s5\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:26.595052 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.595009    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/964d70b0-09d8-46d7-98fa-d034aa975096-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-rz8s5\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:26.597061 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.597039    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-rz8s5\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:26.597470 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.597443    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/964d70b0-09d8-46d7-98fa-d034aa975096-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-rz8s5\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:26.598661 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.598636    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"]
Oct 30 21:44:26.653122 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.653063    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/964d70b0-09d8-46d7-98fa-d034aa975096-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-rz8s5\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:26.653122 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.653101    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27pct\" (UniqueName: \"kubernetes.io/projected/964d70b0-09d8-46d7-98fa-d034aa975096-kube-api-access-27pct\") pod \"maas-default-gateway-openshift-default-74854d8ccf-rz8s5\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:26.709429 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:26.709405    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:27.134293 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:27.134257    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"]
Oct 30 21:44:27.136491 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:44:27.136469    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod964d70b0_09d8_46d7_98fa_d034aa975096.slice/crio-556fd3cceef398ebb259fa9c67ae3a2b3f4bf47ead1aa875f8d9bbc163b5643f WatchSource:0}: Error finding container 556fd3cceef398ebb259fa9c67ae3a2b3f4bf47ead1aa875f8d9bbc163b5643f: Status 404 returned error can't find the container with id 556fd3cceef398ebb259fa9c67ae3a2b3f4bf47ead1aa875f8d9bbc163b5643f
Oct 30 21:44:27.138735 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:27.138707    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:44:27.138809 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:27.138766    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:44:27.138809 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:27.138798    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:44:27.867045 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:27.867007    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5" event={"ID":"964d70b0-09d8-46d7-98fa-d034aa975096","Type":"ContainerStarted","Data":"3c32d0925d2cc3dd4b6e41e8c41b544a80ccecd94b07003d32319338d0174695"}
Oct 30 21:44:27.867499 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:27.867076    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5" event={"ID":"964d70b0-09d8-46d7-98fa-d034aa975096","Type":"ContainerStarted","Data":"556fd3cceef398ebb259fa9c67ae3a2b3f4bf47ead1aa875f8d9bbc163b5643f"}
Oct 30 21:44:28.710129 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:28.710100    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:28.711202 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:28.711184    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-rz8s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 30 21:44:28.711265 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:28.711228    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5" podUID="964d70b0-09d8-46d7-98fa-d034aa975096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 30 21:44:29.710472 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:29.710437    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-rz8s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 30 21:44:29.710854 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:29.710515    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5" podUID="964d70b0-09d8-46d7-98fa-d034aa975096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 30 21:44:30.710179 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:30.710133    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-rz8s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 30 21:44:30.710366 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:30.710198    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5" podUID="964d70b0-09d8-46d7-98fa-d034aa975096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 30 21:44:31.710300 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:31.710264    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-rz8s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 30 21:44:31.710705 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:31.710313    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5" podUID="964d70b0-09d8-46d7-98fa-d034aa975096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 30 21:44:32.710107 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:32.710067    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-rz8s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 30 21:44:32.710299 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:32.710140    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5" podUID="964d70b0-09d8-46d7-98fa-d034aa975096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 30 21:44:33.710020 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:33.709983    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-rz8s5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Oct 30 21:44:33.710440 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:33.710067    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5" podUID="964d70b0-09d8-46d7-98fa-d034aa975096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Oct 30 21:44:35.330895 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:35.330812    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:35.331327 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:35.331207    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:35.332145 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:35.332125    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:44:35.485273 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:44:35.485220    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5" podStartSLOduration=9.485205202 podStartE2EDuration="9.485205202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:27.995692618 +0000 UTC m=+662.470196597" watchObservedRunningTime="2026-10-30 21:44:35.485205202 +0000 UTC m=+669.959709182"
Oct 30 21:44:47.167081 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:44:47.167041    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:45:17.166760 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:45:17.166722    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:45:47.165982 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:45:47.165943    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:45:56.153915 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:45:56.153878    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx"]
Oct 30 21:45:56.157580 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:45:56.157559    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx"
Oct 30 21:45:56.210681 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:45:56.210647    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-bc009f\""
Oct 30 21:45:56.223444 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:45:56.223318    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eab3f328-6507-45d0-abbe-0da0ae5fa74f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx\" (UID: \"eab3f328-6507-45d0-abbe-0da0ae5fa74f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx"
Oct 30 21:45:56.223831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:45:56.223775    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmc7v\" (UniqueName: \"kubernetes.io/projected/eab3f328-6507-45d0-abbe-0da0ae5fa74f-kube-api-access-mmc7v\") pod \"payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx\" (UID: \"eab3f328-6507-45d0-abbe-0da0ae5fa74f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx"
Oct 30 21:45:56.260004 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:45:56.259975    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-bc009f-dockercfg-s5llr\""
Oct 30 21:45:56.261672 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:45:56.261650    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx"]
Oct 30 21:45:56.325313 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:45:56.325281    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmc7v\" (UniqueName: \"kubernetes.io/projected/eab3f328-6507-45d0-abbe-0da0ae5fa74f-kube-api-access-mmc7v\") pod \"payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx\" (UID: \"eab3f328-6507-45d0-abbe-0da0ae5fa74f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx"
Oct 30 21:45:56.325506 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:45:56.325406    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eab3f328-6507-45d0-abbe-0da0ae5fa74f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx\" (UID: \"eab3f328-6507-45d0-abbe-0da0ae5fa74f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx"
Oct 30 21:45:56.326133 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:45:56.326112    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eab3f328-6507-45d0-abbe-0da0ae5fa74f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx\" (UID: \"eab3f328-6507-45d0-abbe-0da0ae5fa74f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx"
Oct 30 21:45:56.382290 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:45:56.382261    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmc7v\" (UniqueName: \"kubernetes.io/projected/eab3f328-6507-45d0-abbe-0da0ae5fa74f-kube-api-access-mmc7v\") pod \"payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx\" (UID: \"eab3f328-6507-45d0-abbe-0da0ae5fa74f\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx"
Oct 30 21:45:56.470499 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:45:56.470422    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx"
Oct 30 21:45:56.848154 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:45:56.848124    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx"]
Oct 30 21:45:56.850833 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:45:56.850801    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeab3f328_6507_45d0_abbe_0da0ae5fa74f.slice/crio-78db747b591502dec16933611edb166b0dd6e238d2208366cf8a8f6ce0a19b11 WatchSource:0}: Error finding container 78db747b591502dec16933611edb166b0dd6e238d2208366cf8a8f6ce0a19b11: Status 404 returned error can't find the container with id 78db747b591502dec16933611edb166b0dd6e238d2208366cf8a8f6ce0a19b11
Oct 30 21:45:56.852847 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:45:56.852828    5087 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:45:57.287303 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:45:57.287265    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx" event={"ID":"eab3f328-6507-45d0-abbe-0da0ae5fa74f","Type":"ContainerStarted","Data":"78db747b591502dec16933611edb166b0dd6e238d2208366cf8a8f6ce0a19b11"}
Oct 30 21:45:58.695116 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:45:58.695075    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh"]
Oct 30 21:45:58.699120 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:45:58.699093    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh"
Oct 30 21:45:58.750052 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:45:58.749732    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf529\" (UniqueName: \"kubernetes.io/projected/ad32b3aa-9229-422b-b7d5-6d2bd8531138-kube-api-access-gf529\") pod \"payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh\" (UID: \"ad32b3aa-9229-422b-b7d5-6d2bd8531138\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh"
Oct 30 21:45:58.750242 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:45:58.750064    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad32b3aa-9229-422b-b7d5-6d2bd8531138-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh\" (UID: \"ad32b3aa-9229-422b-b7d5-6d2bd8531138\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh"
Oct 30 21:45:58.750691 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:45:58.750655    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-9b51e1\""
Oct 30 21:45:58.800335 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:45:58.800287    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-9b51e1-dockercfg-mqg58\""
Oct 30 21:45:58.802551 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:45:58.802524    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh"]
Oct 30 21:45:58.850799 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:45:58.850767    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad32b3aa-9229-422b-b7d5-6d2bd8531138-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh\" (UID: \"ad32b3aa-9229-422b-b7d5-6d2bd8531138\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh"
Oct 30 21:45:58.850984 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:45:58.850834    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf529\" (UniqueName: \"kubernetes.io/projected/ad32b3aa-9229-422b-b7d5-6d2bd8531138-kube-api-access-gf529\") pod \"payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh\" (UID: \"ad32b3aa-9229-422b-b7d5-6d2bd8531138\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh"
Oct 30 21:45:58.851887 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:45:58.851855    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad32b3aa-9229-422b-b7d5-6d2bd8531138-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh\" (UID: \"ad32b3aa-9229-422b-b7d5-6d2bd8531138\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh"
Oct 30 21:45:58.908504 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:45:58.908468    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf529\" (UniqueName: \"kubernetes.io/projected/ad32b3aa-9229-422b-b7d5-6d2bd8531138-kube-api-access-gf529\") pod \"payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh\" (UID: \"ad32b3aa-9229-422b-b7d5-6d2bd8531138\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh"
Oct 30 21:45:59.013885 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:45:59.013847    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh"
Oct 30 21:45:59.404679 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:45:59.404649    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh"]
Oct 30 21:45:59.414250 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:45:59.414214    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad32b3aa_9229_422b_b7d5_6d2bd8531138.slice/crio-77352ab3ee8fe4d25188be6b1a223bc346344e92f57234401a97191ee47d5ffd WatchSource:0}: Error finding container 77352ab3ee8fe4d25188be6b1a223bc346344e92f57234401a97191ee47d5ffd: Status 404 returned error can't find the container with id 77352ab3ee8fe4d25188be6b1a223bc346344e92f57234401a97191ee47d5ffd
Oct 30 21:46:00.306227 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:00.306197    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh" event={"ID":"ad32b3aa-9229-422b-b7d5-6d2bd8531138","Type":"ContainerStarted","Data":"77352ab3ee8fe4d25188be6b1a223bc346344e92f57234401a97191ee47d5ffd"}
Oct 30 21:46:00.308145 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:00.308110    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx" event={"ID":"eab3f328-6507-45d0-abbe-0da0ae5fa74f","Type":"ContainerStarted","Data":"94b18d820afc4bdbd2c6ba390932c4b8b6fd9c730714a56bc2c57bac7b781957"}
Oct 30 21:46:00.308262 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:00.308246    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx"
Oct 30 21:46:00.430200 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:00.430151    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx" podStartSLOduration=1.593652063 podStartE2EDuration="4.430135793s" totalImagesPullingTime="2.83648373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:00.428716951 +0000 UTC m=+754.903220932" watchObservedRunningTime="2026-10-30 21:46:00.430135793 +0000 UTC m=+754.904639771"
Oct 30 21:46:01.038833 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:01.038803    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v"]
Oct 30 21:46:01.042461 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:01.042445    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v"
Oct 30 21:46:01.098915 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:01.098890    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-dda9f5\""
Oct 30 21:46:01.099125 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:01.098956    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-dda9f5-dockercfg-lxvfs\""
Oct 30 21:46:01.153746 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:01.153692    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v"]
Oct 30 21:46:01.176271 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:01.176237    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a9407e-afe4-4e0b-af28-20226fb487c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v\" (UID: \"88a9407e-afe4-4e0b-af28-20226fb487c0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v"
Oct 30 21:46:01.176491 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:01.176300    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l827s\" (UniqueName: \"kubernetes.io/projected/88a9407e-afe4-4e0b-af28-20226fb487c0-kube-api-access-l827s\") pod \"payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v\" (UID: \"88a9407e-afe4-4e0b-af28-20226fb487c0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v"
Oct 30 21:46:01.180226 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:01.180197    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw"]
Oct 30 21:46:01.183925 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:01.183905    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw"
Oct 30 21:46:01.241710 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:01.241677    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw"]
Oct 30 21:46:01.277010 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:01.276982    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7djzj\" (UniqueName: \"kubernetes.io/projected/14ebb851-d1f0-43a8-8351-9fbe3e5a4e71-kube-api-access-7djzj\") pod \"payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw\" (UID: \"14ebb851-d1f0-43a8-8351-9fbe3e5a4e71\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw"
Oct 30 21:46:01.277236 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:01.277027    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14ebb851-d1f0-43a8-8351-9fbe3e5a4e71-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw\" (UID: \"14ebb851-d1f0-43a8-8351-9fbe3e5a4e71\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw"
Oct 30 21:46:01.277236 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:01.277174    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a9407e-afe4-4e0b-af28-20226fb487c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v\" (UID: \"88a9407e-afe4-4e0b-af28-20226fb487c0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v"
Oct 30 21:46:01.277365 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:01.277237    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l827s\" (UniqueName: \"kubernetes.io/projected/88a9407e-afe4-4e0b-af28-20226fb487c0-kube-api-access-l827s\") pod \"payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v\" (UID: \"88a9407e-afe4-4e0b-af28-20226fb487c0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v"
Oct 30 21:46:01.277922 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:01.277901    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a9407e-afe4-4e0b-af28-20226fb487c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v\" (UID: \"88a9407e-afe4-4e0b-af28-20226fb487c0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v"
Oct 30 21:46:01.314907 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:01.314831    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh" event={"ID":"ad32b3aa-9229-422b-b7d5-6d2bd8531138","Type":"ContainerStarted","Data":"6eb8804136865c7eba0d9d31fedceca7c9ce278972d137ea5c02656a9a093f36"}
Oct 30 21:46:01.315309 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:01.315071    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh"
Oct 30 21:46:01.334818 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:01.334788    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l827s\" (UniqueName: \"kubernetes.io/projected/88a9407e-afe4-4e0b-af28-20226fb487c0-kube-api-access-l827s\") pod \"payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v\" (UID: \"88a9407e-afe4-4e0b-af28-20226fb487c0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v"
Oct 30 21:46:01.355433 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:01.355408    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v"
Oct 30 21:46:01.377910 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:01.377884    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7djzj\" (UniqueName: \"kubernetes.io/projected/14ebb851-d1f0-43a8-8351-9fbe3e5a4e71-kube-api-access-7djzj\") pod \"payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw\" (UID: \"14ebb851-d1f0-43a8-8351-9fbe3e5a4e71\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw"
Oct 30 21:46:01.378994 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:01.377952    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14ebb851-d1f0-43a8-8351-9fbe3e5a4e71-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw\" (UID: \"14ebb851-d1f0-43a8-8351-9fbe3e5a4e71\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw"
Oct 30 21:46:01.379050 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:01.378988    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14ebb851-d1f0-43a8-8351-9fbe3e5a4e71-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw\" (UID: \"14ebb851-d1f0-43a8-8351-9fbe3e5a4e71\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw"
Oct 30 21:46:01.454389 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:01.444373    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh" podStartSLOduration=2.6026402060000002 podStartE2EDuration="3.444336415s" totalImagesPullingTime="841.696209ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:01.44068176 +0000 UTC m=+755.915185765" watchObservedRunningTime="2026-10-30 21:46:01.444336415 +0000 UTC m=+755.918840393"
Oct 30 21:46:01.454389 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:01.451867    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7djzj\" (UniqueName: \"kubernetes.io/projected/14ebb851-d1f0-43a8-8351-9fbe3e5a4e71-kube-api-access-7djzj\") pod \"payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw\" (UID: \"14ebb851-d1f0-43a8-8351-9fbe3e5a4e71\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw"
Oct 30 21:46:01.497687 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:01.497655    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw"
Oct 30 21:46:01.727579 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:01.727556    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v"]
Oct 30 21:46:01.729257 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:46:01.729231    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88a9407e_afe4_4e0b_af28_20226fb487c0.slice/crio-663ca1c8ad544c23d90bdc4a7fab70687736420ae5cfcf69ef394fe0c6707351 WatchSource:0}: Error finding container 663ca1c8ad544c23d90bdc4a7fab70687736420ae5cfcf69ef394fe0c6707351: Status 404 returned error can't find the container with id 663ca1c8ad544c23d90bdc4a7fab70687736420ae5cfcf69ef394fe0c6707351
Oct 30 21:46:01.896103 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:01.896081    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw"]
Oct 30 21:46:01.897982 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:46:01.897957    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14ebb851_d1f0_43a8_8351_9fbe3e5a4e71.slice/crio-77b52e76f9a3cec4f2e6ef708417b86da7434de106585a15f039788fb1f6351b WatchSource:0}: Error finding container 77b52e76f9a3cec4f2e6ef708417b86da7434de106585a15f039788fb1f6351b: Status 404 returned error can't find the container with id 77b52e76f9a3cec4f2e6ef708417b86da7434de106585a15f039788fb1f6351b
Oct 30 21:46:02.321429 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:02.321390    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v" event={"ID":"88a9407e-afe4-4e0b-af28-20226fb487c0","Type":"ContainerStarted","Data":"bf880dd59f6d51e81b9cf036e97aae707e3315cb42adf4b9002491dccd609583"}
Oct 30 21:46:02.321894 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:02.321450    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v" event={"ID":"88a9407e-afe4-4e0b-af28-20226fb487c0","Type":"ContainerStarted","Data":"663ca1c8ad544c23d90bdc4a7fab70687736420ae5cfcf69ef394fe0c6707351"}
Oct 30 21:46:02.321894 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:02.321575    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v"
Oct 30 21:46:02.323104 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:02.323074    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw" event={"ID":"14ebb851-d1f0-43a8-8351-9fbe3e5a4e71","Type":"ContainerStarted","Data":"e01ff47a8e18f1b2569c9ab88e80f25759536c33cea399cf3d8e7178fc6cdcf9"}
Oct 30 21:46:02.323259 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:02.323153    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw" event={"ID":"14ebb851-d1f0-43a8-8351-9fbe3e5a4e71","Type":"ContainerStarted","Data":"77b52e76f9a3cec4f2e6ef708417b86da7434de106585a15f039788fb1f6351b"}
Oct 30 21:46:02.323259 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:02.323195    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw"
Oct 30 21:46:02.435152 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:02.435102    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v" podStartSLOduration=2.435088889 podStartE2EDuration="2.435088889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:02.433728994 +0000 UTC m=+756.908232976" watchObservedRunningTime="2026-10-30 21:46:02.435088889 +0000 UTC m=+756.909592867"
Oct 30 21:46:02.550843 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:02.550790    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw" podStartSLOduration=1.550774055 podStartE2EDuration="1.550774055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:02.54930669 +0000 UTC m=+757.023810677" watchObservedRunningTime="2026-10-30 21:46:02.550774055 +0000 UTC m=+757.025278055"
Oct 30 21:46:11.318976 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:11.318896    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx"
Oct 30 21:46:12.326327 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:12.326301    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh"
Oct 30 21:46:13.332215 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:13.332182    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v"
Oct 30 21:46:13.332616 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:13.332336    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw"
Oct 30 21:46:17.166875 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:46:17.166834    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:46:29.740392 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:29.740359    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"]
Oct 30 21:46:29.744388 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:29.744368    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:46:29.835906 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:29.835873    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-model-cache\") pod \"simulator-w6-557708-kserve-54b6d54c8f-8nsj7\" (UID: \"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf\") " pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:46:29.836087 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:29.835935    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dqhp\" (UniqueName: \"kubernetes.io/projected/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-kube-api-access-8dqhp\") pod \"simulator-w6-557708-kserve-54b6d54c8f-8nsj7\" (UID: \"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf\") " pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:46:29.836087 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:29.836067    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-home\") pod \"simulator-w6-557708-kserve-54b6d54c8f-8nsj7\" (UID: \"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf\") " pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:46:29.836205 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:29.836095    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-tls-certs\") pod \"simulator-w6-557708-kserve-54b6d54c8f-8nsj7\" (UID: \"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf\") " pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:46:29.836205 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:29.836122    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-dshm\") pod \"simulator-w6-557708-kserve-54b6d54c8f-8nsj7\" (UID: \"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf\") " pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:46:29.836309 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:29.836244    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-tmp-dir\") pod \"simulator-w6-557708-kserve-54b6d54c8f-8nsj7\" (UID: \"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf\") " pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:46:29.851238 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:29.851211    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-557708\"/\"simulator-w6-557708-kserve-self-signed-certs\""
Oct 30 21:46:29.851440 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:29.851360    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-557708\"/\"openshift-service-ca.crt\""
Oct 30 21:46:29.851440 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:29.851409    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-557708\"/\"kube-root-ca.crt\""
Oct 30 21:46:29.853564 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:29.853544    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"]
Oct 30 21:46:29.906586 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:29.906555    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-557708\"/\"default-dockercfg-g4br6\""
Oct 30 21:46:29.937146 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:29.937114    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-home\") pod \"simulator-w6-557708-kserve-54b6d54c8f-8nsj7\" (UID: \"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf\") " pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:46:29.937387 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:29.937154    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-tls-certs\") pod \"simulator-w6-557708-kserve-54b6d54c8f-8nsj7\" (UID: \"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf\") " pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:46:29.937387 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:29.937180    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-dshm\") pod \"simulator-w6-557708-kserve-54b6d54c8f-8nsj7\" (UID: \"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf\") " pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:46:29.937387 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:29.937208    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-tmp-dir\") pod \"simulator-w6-557708-kserve-54b6d54c8f-8nsj7\" (UID: \"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf\") " pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:46:29.937579 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:29.937474    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-model-cache\") pod \"simulator-w6-557708-kserve-54b6d54c8f-8nsj7\" (UID: \"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf\") " pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:46:29.937634 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:29.937575    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dqhp\" (UniqueName: \"kubernetes.io/projected/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-kube-api-access-8dqhp\") pod \"simulator-w6-557708-kserve-54b6d54c8f-8nsj7\" (UID: \"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf\") " pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:46:29.937634 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:29.937591    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-home\") pod \"simulator-w6-557708-kserve-54b6d54c8f-8nsj7\" (UID: \"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf\") " pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:46:29.938108 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:29.938067    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-tmp-dir\") pod \"simulator-w6-557708-kserve-54b6d54c8f-8nsj7\" (UID: \"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf\") " pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:46:29.938425 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:29.938401    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-model-cache\") pod \"simulator-w6-557708-kserve-54b6d54c8f-8nsj7\" (UID: \"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf\") " pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:46:29.940239 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:29.940213    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-tls-certs\") pod \"simulator-w6-557708-kserve-54b6d54c8f-8nsj7\" (UID: \"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf\") " pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:46:29.940399 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:29.940284    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-dshm\") pod \"simulator-w6-557708-kserve-54b6d54c8f-8nsj7\" (UID: \"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf\") " pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:46:29.997369 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:29.997294    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dqhp\" (UniqueName: \"kubernetes.io/projected/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-kube-api-access-8dqhp\") pod \"simulator-w6-557708-kserve-54b6d54c8f-8nsj7\" (UID: \"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf\") " pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:46:30.066582 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:30.066556    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:46:30.446834 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:30.446295    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"]
Oct 30 21:46:30.449240 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:46:30.449215    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7aea1c0_4e06_4f42_8a0a_ef82ae5e44bf.slice/crio-a99c8eba11020044b14e9d68bb017e04070612308449069560e7597d73b7d89e WatchSource:0}: Error finding container a99c8eba11020044b14e9d68bb017e04070612308449069560e7597d73b7d89e: Status 404 returned error can't find the container with id a99c8eba11020044b14e9d68bb017e04070612308449069560e7597d73b7d89e
Oct 30 21:46:30.467449 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:30.467422    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" event={"ID":"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf","Type":"ContainerStarted","Data":"a99c8eba11020044b14e9d68bb017e04070612308449069560e7597d73b7d89e"}
Oct 30 21:46:33.491686 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:33.491643    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" event={"ID":"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf","Type":"ContainerStarted","Data":"d9536199e5a33bbdc75f3aeca78b92be08e5befec60873054fcbd748aae142d5"}
Oct 30 21:46:33.609192 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:33.609131    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" podStartSLOduration=2.40962666 podStartE2EDuration="4.609112157s" totalImagesPullingTime="2.199485497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:33.607865016 +0000 UTC m=+788.082369006" watchObservedRunningTime="2026-10-30 21:46:33.609112157 +0000 UTC m=+788.083616139"
Oct 30 21:46:34.067063 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:34.067017    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:46:40.067226 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:40.067187    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:46:40.079702 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:40.079673    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:46:40.542923 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:40.542896    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:46:47.166294 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:46:47.166260    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:46:57.475660 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:57.475617    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"]
Oct 30 21:46:57.478922 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:57.478901    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:46:57.531206 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:57.531181    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-80c366\"/\"openshift-service-ca.crt\""
Oct 30 21:46:57.559894 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:57.559867    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-80c366\"/\"kube-root-ca.crt\""
Oct 30 21:46:57.580522 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:57.580486    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-80c366\"/\"simulator-w1-80c366-kserve-self-signed-certs\""
Oct 30 21:46:57.580522 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:57.580513    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-80c366\"/\"default-dockercfg-xrtxm\""
Oct 30 21:46:57.583593 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:57.583571    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"]
Oct 30 21:46:57.622017 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:57.621987    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0116-60ed-4471-aba6-a2cc23eb8480-tmp-dir\") pod \"simulator-w1-80c366-kserve-5f67bc7b86-f9rsl\" (UID: \"ca9b0116-60ed-4471-aba6-a2cc23eb8480\") " pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:46:57.622157 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:57.622040    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0116-60ed-4471-aba6-a2cc23eb8480-model-cache\") pod \"simulator-w1-80c366-kserve-5f67bc7b86-f9rsl\" (UID: \"ca9b0116-60ed-4471-aba6-a2cc23eb8480\") " pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:46:57.622199 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:57.622181    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0116-60ed-4471-aba6-a2cc23eb8480-dshm\") pod \"simulator-w1-80c366-kserve-5f67bc7b86-f9rsl\" (UID: \"ca9b0116-60ed-4471-aba6-a2cc23eb8480\") " pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:46:57.622233 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:57.622222    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0116-60ed-4471-aba6-a2cc23eb8480-home\") pod \"simulator-w1-80c366-kserve-5f67bc7b86-f9rsl\" (UID: \"ca9b0116-60ed-4471-aba6-a2cc23eb8480\") " pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:46:57.622321 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:57.622301    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdlmc\" (UniqueName: \"kubernetes.io/projected/ca9b0116-60ed-4471-aba6-a2cc23eb8480-kube-api-access-tdlmc\") pod \"simulator-w1-80c366-kserve-5f67bc7b86-f9rsl\" (UID: \"ca9b0116-60ed-4471-aba6-a2cc23eb8480\") " pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:46:57.622406 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:57.622367    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca9b0116-60ed-4471-aba6-a2cc23eb8480-tls-certs\") pod \"simulator-w1-80c366-kserve-5f67bc7b86-f9rsl\" (UID: \"ca9b0116-60ed-4471-aba6-a2cc23eb8480\") " pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:46:57.723665 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:57.723633    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0116-60ed-4471-aba6-a2cc23eb8480-home\") pod \"simulator-w1-80c366-kserve-5f67bc7b86-f9rsl\" (UID: \"ca9b0116-60ed-4471-aba6-a2cc23eb8480\") " pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:46:57.723665 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:57.723672    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdlmc\" (UniqueName: \"kubernetes.io/projected/ca9b0116-60ed-4471-aba6-a2cc23eb8480-kube-api-access-tdlmc\") pod \"simulator-w1-80c366-kserve-5f67bc7b86-f9rsl\" (UID: \"ca9b0116-60ed-4471-aba6-a2cc23eb8480\") " pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:46:57.723877 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:57.723693    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca9b0116-60ed-4471-aba6-a2cc23eb8480-tls-certs\") pod \"simulator-w1-80c366-kserve-5f67bc7b86-f9rsl\" (UID: \"ca9b0116-60ed-4471-aba6-a2cc23eb8480\") " pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:46:57.723877 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:57.723742    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0116-60ed-4471-aba6-a2cc23eb8480-tmp-dir\") pod \"simulator-w1-80c366-kserve-5f67bc7b86-f9rsl\" (UID: \"ca9b0116-60ed-4471-aba6-a2cc23eb8480\") " pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:46:57.723877 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:57.723776    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0116-60ed-4471-aba6-a2cc23eb8480-model-cache\") pod \"simulator-w1-80c366-kserve-5f67bc7b86-f9rsl\" (UID: \"ca9b0116-60ed-4471-aba6-a2cc23eb8480\") " pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:46:57.723877 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:57.723855    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0116-60ed-4471-aba6-a2cc23eb8480-dshm\") pod \"simulator-w1-80c366-kserve-5f67bc7b86-f9rsl\" (UID: \"ca9b0116-60ed-4471-aba6-a2cc23eb8480\") " pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:46:57.724097 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:57.724073    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0116-60ed-4471-aba6-a2cc23eb8480-home\") pod \"simulator-w1-80c366-kserve-5f67bc7b86-f9rsl\" (UID: \"ca9b0116-60ed-4471-aba6-a2cc23eb8480\") " pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:46:57.724134 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:57.724092    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0116-60ed-4471-aba6-a2cc23eb8480-tmp-dir\") pod \"simulator-w1-80c366-kserve-5f67bc7b86-f9rsl\" (UID: \"ca9b0116-60ed-4471-aba6-a2cc23eb8480\") " pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:46:57.724177 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:57.724156    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0116-60ed-4471-aba6-a2cc23eb8480-model-cache\") pod \"simulator-w1-80c366-kserve-5f67bc7b86-f9rsl\" (UID: \"ca9b0116-60ed-4471-aba6-a2cc23eb8480\") " pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:46:57.726259 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:57.726211    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0116-60ed-4471-aba6-a2cc23eb8480-dshm\") pod \"simulator-w1-80c366-kserve-5f67bc7b86-f9rsl\" (UID: \"ca9b0116-60ed-4471-aba6-a2cc23eb8480\") " pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:46:57.726391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:57.726376    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca9b0116-60ed-4471-aba6-a2cc23eb8480-tls-certs\") pod \"simulator-w1-80c366-kserve-5f67bc7b86-f9rsl\" (UID: \"ca9b0116-60ed-4471-aba6-a2cc23eb8480\") " pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:46:57.781257 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:57.781215    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdlmc\" (UniqueName: \"kubernetes.io/projected/ca9b0116-60ed-4471-aba6-a2cc23eb8480-kube-api-access-tdlmc\") pod \"simulator-w1-80c366-kserve-5f67bc7b86-f9rsl\" (UID: \"ca9b0116-60ed-4471-aba6-a2cc23eb8480\") " pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:46:57.790053 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:57.790027    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:46:58.168693 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:58.168654    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"]
Oct 30 21:46:58.171448 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:46:58.171426    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca9b0116_60ed_4471_aba6_a2cc23eb8480.slice/crio-cf8a132c6f8f63e067c9a2ff2641e33b7d6738e36eb7d8ced103154aa318e7c1 WatchSource:0}: Error finding container cf8a132c6f8f63e067c9a2ff2641e33b7d6738e36eb7d8ced103154aa318e7c1: Status 404 returned error can't find the container with id cf8a132c6f8f63e067c9a2ff2641e33b7d6738e36eb7d8ced103154aa318e7c1
Oct 30 21:46:58.639707 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:58.639672    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" event={"ID":"ca9b0116-60ed-4471-aba6-a2cc23eb8480","Type":"ContainerStarted","Data":"8d2988189c44731c4b36c5f22dd19b76e057a34260654fa670b473456dd838a4"}
Oct 30 21:46:58.640206 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:58.639725    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" event={"ID":"ca9b0116-60ed-4471-aba6-a2cc23eb8480","Type":"ContainerStarted","Data":"cf8a132c6f8f63e067c9a2ff2641e33b7d6738e36eb7d8ced103154aa318e7c1"}
Oct 30 21:46:58.756671 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:58.756604    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" podStartSLOduration=1.75658396 podStartE2EDuration="1.75658396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:58.754512586 +0000 UTC m=+813.229016565" watchObservedRunningTime="2026-10-30 21:46:58.75658396 +0000 UTC m=+813.231087942"
Oct 30 21:46:58.790561 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:46:58.790530    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:47:07.790362 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:07.790323    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:47:07.802848 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:07.802822    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:47:08.700304 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:08.700275    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:47:14.262146 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:14.262114    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"]
Oct 30 21:47:14.265830 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:14.265796    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:47:14.329020 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:14.328963    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c79cb5\"/\"openshift-service-ca.crt\""
Oct 30 21:47:14.368478 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:14.368445    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c79cb5\"/\"default-dockercfg-sbmpt\""
Oct 30 21:47:14.368478 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:14.368447    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c79cb5\"/\"kube-root-ca.crt\""
Oct 30 21:47:14.368877 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:14.368861    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c79cb5\"/\"simulator-w2-c79cb5-kserve-self-signed-certs\""
Oct 30 21:47:14.371384 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:14.371337    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"]
Oct 30 21:47:14.380436 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:14.380412    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r76w\" (UniqueName: \"kubernetes.io/projected/06f6da2e-4915-493e-b4fe-f06c2b8612e5-kube-api-access-5r76w\") pod \"simulator-w2-c79cb5-kserve-fbbb584fb-jscm9\" (UID: \"06f6da2e-4915-493e-b4fe-f06c2b8612e5\") " pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:47:14.380544 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:14.380446    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06f6da2e-4915-493e-b4fe-f06c2b8612e5-model-cache\") pod \"simulator-w2-c79cb5-kserve-fbbb584fb-jscm9\" (UID: \"06f6da2e-4915-493e-b4fe-f06c2b8612e5\") " pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:47:14.380620 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:14.380603    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06f6da2e-4915-493e-b4fe-f06c2b8612e5-tls-certs\") pod \"simulator-w2-c79cb5-kserve-fbbb584fb-jscm9\" (UID: \"06f6da2e-4915-493e-b4fe-f06c2b8612e5\") " pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:47:14.380664 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:14.380638    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06f6da2e-4915-493e-b4fe-f06c2b8612e5-home\") pod \"simulator-w2-c79cb5-kserve-fbbb584fb-jscm9\" (UID: \"06f6da2e-4915-493e-b4fe-f06c2b8612e5\") " pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:47:14.380716 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:14.380674    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f6da2e-4915-493e-b4fe-f06c2b8612e5-tmp-dir\") pod \"simulator-w2-c79cb5-kserve-fbbb584fb-jscm9\" (UID: \"06f6da2e-4915-493e-b4fe-f06c2b8612e5\") " pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:47:14.380788 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:14.380777    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06f6da2e-4915-493e-b4fe-f06c2b8612e5-dshm\") pod \"simulator-w2-c79cb5-kserve-fbbb584fb-jscm9\" (UID: \"06f6da2e-4915-493e-b4fe-f06c2b8612e5\") " pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:47:14.481623 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:14.481589    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06f6da2e-4915-493e-b4fe-f06c2b8612e5-tls-certs\") pod \"simulator-w2-c79cb5-kserve-fbbb584fb-jscm9\" (UID: \"06f6da2e-4915-493e-b4fe-f06c2b8612e5\") " pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:47:14.481811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:14.481639    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06f6da2e-4915-493e-b4fe-f06c2b8612e5-home\") pod \"simulator-w2-c79cb5-kserve-fbbb584fb-jscm9\" (UID: \"06f6da2e-4915-493e-b4fe-f06c2b8612e5\") " pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:47:14.481811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:14.481687    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f6da2e-4915-493e-b4fe-f06c2b8612e5-tmp-dir\") pod \"simulator-w2-c79cb5-kserve-fbbb584fb-jscm9\" (UID: \"06f6da2e-4915-493e-b4fe-f06c2b8612e5\") " pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:47:14.481811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:14.481756    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06f6da2e-4915-493e-b4fe-f06c2b8612e5-dshm\") pod \"simulator-w2-c79cb5-kserve-fbbb584fb-jscm9\" (UID: \"06f6da2e-4915-493e-b4fe-f06c2b8612e5\") " pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:47:14.481811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:14.481794    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r76w\" (UniqueName: \"kubernetes.io/projected/06f6da2e-4915-493e-b4fe-f06c2b8612e5-kube-api-access-5r76w\") pod \"simulator-w2-c79cb5-kserve-fbbb584fb-jscm9\" (UID: \"06f6da2e-4915-493e-b4fe-f06c2b8612e5\") " pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:47:14.482000 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:14.481820    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06f6da2e-4915-493e-b4fe-f06c2b8612e5-model-cache\") pod \"simulator-w2-c79cb5-kserve-fbbb584fb-jscm9\" (UID: \"06f6da2e-4915-493e-b4fe-f06c2b8612e5\") " pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:47:14.482183 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:14.482155    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06f6da2e-4915-493e-b4fe-f06c2b8612e5-home\") pod \"simulator-w2-c79cb5-kserve-fbbb584fb-jscm9\" (UID: \"06f6da2e-4915-493e-b4fe-f06c2b8612e5\") " pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:47:14.482276 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:14.482201    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f6da2e-4915-493e-b4fe-f06c2b8612e5-tmp-dir\") pod \"simulator-w2-c79cb5-kserve-fbbb584fb-jscm9\" (UID: \"06f6da2e-4915-493e-b4fe-f06c2b8612e5\") " pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:47:14.482320 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:14.482286    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06f6da2e-4915-493e-b4fe-f06c2b8612e5-model-cache\") pod \"simulator-w2-c79cb5-kserve-fbbb584fb-jscm9\" (UID: \"06f6da2e-4915-493e-b4fe-f06c2b8612e5\") " pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:47:14.484204 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:14.484182    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06f6da2e-4915-493e-b4fe-f06c2b8612e5-dshm\") pod \"simulator-w2-c79cb5-kserve-fbbb584fb-jscm9\" (UID: \"06f6da2e-4915-493e-b4fe-f06c2b8612e5\") " pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:47:14.484515 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:14.484484    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06f6da2e-4915-493e-b4fe-f06c2b8612e5-tls-certs\") pod \"simulator-w2-c79cb5-kserve-fbbb584fb-jscm9\" (UID: \"06f6da2e-4915-493e-b4fe-f06c2b8612e5\") " pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:47:14.546008 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:14.545932    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r76w\" (UniqueName: \"kubernetes.io/projected/06f6da2e-4915-493e-b4fe-f06c2b8612e5-kube-api-access-5r76w\") pod \"simulator-w2-c79cb5-kserve-fbbb584fb-jscm9\" (UID: \"06f6da2e-4915-493e-b4fe-f06c2b8612e5\") " pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:47:14.578951 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:14.578926    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:47:14.972963 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:14.972935    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"]
Oct 30 21:47:14.975713 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:47:14.975673    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06f6da2e_4915_493e_b4fe_f06c2b8612e5.slice/crio-e6ebd25008f7d790d1e3ac1d08925640d6c1da126076a1731ecbef60e5046c41 WatchSource:0}: Error finding container e6ebd25008f7d790d1e3ac1d08925640d6c1da126076a1731ecbef60e5046c41: Status 404 returned error can't find the container with id e6ebd25008f7d790d1e3ac1d08925640d6c1da126076a1731ecbef60e5046c41
Oct 30 21:47:15.725491 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:15.725453    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" event={"ID":"06f6da2e-4915-493e-b4fe-f06c2b8612e5","Type":"ContainerStarted","Data":"5c769f0a046092765d98c06e4d5e669273856ac9f991fd7b9c829bd78174769f"}
Oct 30 21:47:15.725867 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:15.725499    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" event={"ID":"06f6da2e-4915-493e-b4fe-f06c2b8612e5","Type":"ContainerStarted","Data":"e6ebd25008f7d790d1e3ac1d08925640d6c1da126076a1731ecbef60e5046c41"}
Oct 30 21:47:15.846633 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:15.846578    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podStartSLOduration=1.8465612999999999 podStartE2EDuration="1.8465613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:15.84523338 +0000 UTC m=+830.319737361" watchObservedRunningTime="2026-10-30 21:47:15.8465613 +0000 UTC m=+830.321065279"
Oct 30 21:47:16.579721 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:16.579690    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:47:17.168197 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:47:17.168153    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:47:24.579146 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:24.579098    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:47:24.592456 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:24.592434    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:47:24.790586 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:24.790554    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:47:25.481828 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:25.481793    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"]
Oct 30 21:47:25.485707 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:25.485681    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:47:25.547263 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:25.547238    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-557708\"/\"premium-w6-557708-kserve-self-signed-certs\""
Oct 30 21:47:25.551193 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:25.551167    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"]
Oct 30 21:47:25.590090 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:25.590051    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/699a4ca0-a268-4109-a8f8-dde6bf132689-tmp-dir\") pod \"premium-w6-557708-kserve-5575f79844-cr9xc\" (UID: \"699a4ca0-a268-4109-a8f8-dde6bf132689\") " pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:47:25.590593 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:25.590097    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/699a4ca0-a268-4109-a8f8-dde6bf132689-dshm\") pod \"premium-w6-557708-kserve-5575f79844-cr9xc\" (UID: \"699a4ca0-a268-4109-a8f8-dde6bf132689\") " pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:47:25.590593 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:25.590214    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhp7n\" (UniqueName: \"kubernetes.io/projected/699a4ca0-a268-4109-a8f8-dde6bf132689-kube-api-access-lhp7n\") pod \"premium-w6-557708-kserve-5575f79844-cr9xc\" (UID: \"699a4ca0-a268-4109-a8f8-dde6bf132689\") " pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:47:25.590593 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:25.590323    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/699a4ca0-a268-4109-a8f8-dde6bf132689-home\") pod \"premium-w6-557708-kserve-5575f79844-cr9xc\" (UID: \"699a4ca0-a268-4109-a8f8-dde6bf132689\") " pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:47:25.590593 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:25.590403    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/699a4ca0-a268-4109-a8f8-dde6bf132689-tls-certs\") pod \"premium-w6-557708-kserve-5575f79844-cr9xc\" (UID: \"699a4ca0-a268-4109-a8f8-dde6bf132689\") " pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:47:25.590593 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:25.590441    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/699a4ca0-a268-4109-a8f8-dde6bf132689-model-cache\") pod \"premium-w6-557708-kserve-5575f79844-cr9xc\" (UID: \"699a4ca0-a268-4109-a8f8-dde6bf132689\") " pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:47:25.691265 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:25.691224    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/699a4ca0-a268-4109-a8f8-dde6bf132689-tmp-dir\") pod \"premium-w6-557708-kserve-5575f79844-cr9xc\" (UID: \"699a4ca0-a268-4109-a8f8-dde6bf132689\") " pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:47:25.691457 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:25.691269    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/699a4ca0-a268-4109-a8f8-dde6bf132689-dshm\") pod \"premium-w6-557708-kserve-5575f79844-cr9xc\" (UID: \"699a4ca0-a268-4109-a8f8-dde6bf132689\") " pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:47:25.691457 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:25.691325    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhp7n\" (UniqueName: \"kubernetes.io/projected/699a4ca0-a268-4109-a8f8-dde6bf132689-kube-api-access-lhp7n\") pod \"premium-w6-557708-kserve-5575f79844-cr9xc\" (UID: \"699a4ca0-a268-4109-a8f8-dde6bf132689\") " pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:47:25.691457 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:25.691438    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/699a4ca0-a268-4109-a8f8-dde6bf132689-home\") pod \"premium-w6-557708-kserve-5575f79844-cr9xc\" (UID: \"699a4ca0-a268-4109-a8f8-dde6bf132689\") " pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:47:25.691651 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:25.691485    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/699a4ca0-a268-4109-a8f8-dde6bf132689-tls-certs\") pod \"premium-w6-557708-kserve-5575f79844-cr9xc\" (UID: \"699a4ca0-a268-4109-a8f8-dde6bf132689\") " pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:47:25.691651 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:25.691517    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/699a4ca0-a268-4109-a8f8-dde6bf132689-model-cache\") pod \"premium-w6-557708-kserve-5575f79844-cr9xc\" (UID: \"699a4ca0-a268-4109-a8f8-dde6bf132689\") " pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:47:25.691759 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:25.691665    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/699a4ca0-a268-4109-a8f8-dde6bf132689-tmp-dir\") pod \"premium-w6-557708-kserve-5575f79844-cr9xc\" (UID: \"699a4ca0-a268-4109-a8f8-dde6bf132689\") " pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:47:25.691871 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:25.691850    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/699a4ca0-a268-4109-a8f8-dde6bf132689-model-cache\") pod \"premium-w6-557708-kserve-5575f79844-cr9xc\" (UID: \"699a4ca0-a268-4109-a8f8-dde6bf132689\") " pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:47:25.691951 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:25.691907    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/699a4ca0-a268-4109-a8f8-dde6bf132689-home\") pod \"premium-w6-557708-kserve-5575f79844-cr9xc\" (UID: \"699a4ca0-a268-4109-a8f8-dde6bf132689\") " pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:47:25.693931 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:25.693902    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/699a4ca0-a268-4109-a8f8-dde6bf132689-dshm\") pod \"premium-w6-557708-kserve-5575f79844-cr9xc\" (UID: \"699a4ca0-a268-4109-a8f8-dde6bf132689\") " pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:47:25.744730 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:25.744637    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-557708\"/\"premium-w6-557708-kserve-self-signed-certs\""
Oct 30 21:47:25.755178 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:25.755151    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/699a4ca0-a268-4109-a8f8-dde6bf132689-tls-certs\") pod \"premium-w6-557708-kserve-5575f79844-cr9xc\" (UID: \"699a4ca0-a268-4109-a8f8-dde6bf132689\") " pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:47:25.756566 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:25.756544    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhp7n\" (UniqueName: \"kubernetes.io/projected/699a4ca0-a268-4109-a8f8-dde6bf132689-kube-api-access-lhp7n\") pod \"premium-w6-557708-kserve-5575f79844-cr9xc\" (UID: \"699a4ca0-a268-4109-a8f8-dde6bf132689\") " pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:47:25.803924 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:25.803870    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:47:26.212014 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:26.211986    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"]
Oct 30 21:47:26.212751 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:47:26.212724    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod699a4ca0_a268_4109_a8f8_dde6bf132689.slice/crio-628a197c0fea52dcd7700addc89ef26df239fe285e78a25604acaa7a9ba6d7b2 WatchSource:0}: Error finding container 628a197c0fea52dcd7700addc89ef26df239fe285e78a25604acaa7a9ba6d7b2: Status 404 returned error can't find the container with id 628a197c0fea52dcd7700addc89ef26df239fe285e78a25604acaa7a9ba6d7b2
Oct 30 21:47:26.786413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:26.786378    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" event={"ID":"699a4ca0-a268-4109-a8f8-dde6bf132689","Type":"ContainerStarted","Data":"07a6d7d8fc0834b6f69dec73f8813ac77f2c9e7a297c750c644dbdb75e4b0f18"}
Oct 30 21:47:26.786839 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:26.786423    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" event={"ID":"699a4ca0-a268-4109-a8f8-dde6bf132689","Type":"ContainerStarted","Data":"628a197c0fea52dcd7700addc89ef26df239fe285e78a25604acaa7a9ba6d7b2"}
Oct 30 21:47:26.804689 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:26.804666    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:47:26.914288 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:26.914239    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" podStartSLOduration=1.9142270190000001 podStartE2EDuration="1.914227019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:26.913415506 +0000 UTC m=+841.387919486" watchObservedRunningTime="2026-10-30 21:47:26.914227019 +0000 UTC m=+841.388730995"
Oct 30 21:47:31.738977 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:31.738937    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"]
Oct 30 21:47:31.742889 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:31.742866    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:47:31.851853 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:31.851818    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c3e2d8\"/\"kube-root-ca.crt\""
Oct 30 21:47:31.852004 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:31.851911    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c3e2d8\"/\"simulator-w4-c3e2d8-kserve-self-signed-certs\""
Oct 30 21:47:31.852330 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:31.852314    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c3e2d8\"/\"default-dockercfg-skv5v\""
Oct 30 21:47:31.854028 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:31.853999    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"]
Oct 30 21:47:31.854432 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:31.854413    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b066538d-4d14-4228-b563-bce796a28d0b-home\") pod \"simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj\" (UID: \"b066538d-4d14-4228-b563-bce796a28d0b\") " pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:47:31.854477 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:31.854464    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b066538d-4d14-4228-b563-bce796a28d0b-dshm\") pod \"simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj\" (UID: \"b066538d-4d14-4228-b563-bce796a28d0b\") " pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:47:31.854593 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:31.854579    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gh7z\" (UniqueName: \"kubernetes.io/projected/b066538d-4d14-4228-b563-bce796a28d0b-kube-api-access-8gh7z\") pod \"simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj\" (UID: \"b066538d-4d14-4228-b563-bce796a28d0b\") " pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:47:31.854630 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:31.854604    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b066538d-4d14-4228-b563-bce796a28d0b-tmp-dir\") pod \"simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj\" (UID: \"b066538d-4d14-4228-b563-bce796a28d0b\") " pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:47:31.854730 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:31.854713    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b066538d-4d14-4228-b563-bce796a28d0b-model-cache\") pod \"simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj\" (UID: \"b066538d-4d14-4228-b563-bce796a28d0b\") " pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:47:31.854790 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:31.854753    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b066538d-4d14-4228-b563-bce796a28d0b-tls-certs\") pod \"simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj\" (UID: \"b066538d-4d14-4228-b563-bce796a28d0b\") " pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:47:31.903327 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:31.903303    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-c3e2d8\"/\"openshift-service-ca.crt\""
Oct 30 21:47:31.955729 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:31.955699    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gh7z\" (UniqueName: \"kubernetes.io/projected/b066538d-4d14-4228-b563-bce796a28d0b-kube-api-access-8gh7z\") pod \"simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj\" (UID: \"b066538d-4d14-4228-b563-bce796a28d0b\") " pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:47:31.955889 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:31.955734    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b066538d-4d14-4228-b563-bce796a28d0b-tmp-dir\") pod \"simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj\" (UID: \"b066538d-4d14-4228-b563-bce796a28d0b\") " pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:47:31.955889 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:31.955780    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b066538d-4d14-4228-b563-bce796a28d0b-model-cache\") pod \"simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj\" (UID: \"b066538d-4d14-4228-b563-bce796a28d0b\") " pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:47:31.955889 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:31.955795    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b066538d-4d14-4228-b563-bce796a28d0b-tls-certs\") pod \"simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj\" (UID: \"b066538d-4d14-4228-b563-bce796a28d0b\") " pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:47:31.955889 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:31.955823    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b066538d-4d14-4228-b563-bce796a28d0b-home\") pod \"simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj\" (UID: \"b066538d-4d14-4228-b563-bce796a28d0b\") " pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:47:31.955889 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:31.955872    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b066538d-4d14-4228-b563-bce796a28d0b-dshm\") pod \"simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj\" (UID: \"b066538d-4d14-4228-b563-bce796a28d0b\") " pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:47:31.956204 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:31.956185    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b066538d-4d14-4228-b563-bce796a28d0b-tmp-dir\") pod \"simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj\" (UID: \"b066538d-4d14-4228-b563-bce796a28d0b\") " pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:47:31.956276 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:31.956242    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b066538d-4d14-4228-b563-bce796a28d0b-model-cache\") pod \"simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj\" (UID: \"b066538d-4d14-4228-b563-bce796a28d0b\") " pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:47:31.956393 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:31.956373    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b066538d-4d14-4228-b563-bce796a28d0b-home\") pod \"simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj\" (UID: \"b066538d-4d14-4228-b563-bce796a28d0b\") " pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:47:31.958212 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:31.958189    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b066538d-4d14-4228-b563-bce796a28d0b-dshm\") pod \"simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj\" (UID: \"b066538d-4d14-4228-b563-bce796a28d0b\") " pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:47:31.958618 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:31.958601    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b066538d-4d14-4228-b563-bce796a28d0b-tls-certs\") pod \"simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj\" (UID: \"b066538d-4d14-4228-b563-bce796a28d0b\") " pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:47:32.016501 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:32.016430    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gh7z\" (UniqueName: \"kubernetes.io/projected/b066538d-4d14-4228-b563-bce796a28d0b-kube-api-access-8gh7z\") pod \"simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj\" (UID: \"b066538d-4d14-4228-b563-bce796a28d0b\") " pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:47:32.087839 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:32.087800    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:47:32.472567 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:32.472543    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"]
Oct 30 21:47:32.475015 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:47:32.474986    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb066538d_4d14_4228_b563_bce796a28d0b.slice/crio-e31b0c31e0a5c99f58fcecf65b52e9e103f0f6730bbe396597a61489319d02f6 WatchSource:0}: Error finding container e31b0c31e0a5c99f58fcecf65b52e9e103f0f6730bbe396597a61489319d02f6: Status 404 returned error can't find the container with id e31b0c31e0a5c99f58fcecf65b52e9e103f0f6730bbe396597a61489319d02f6
Oct 30 21:47:32.821991 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:32.821953    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" event={"ID":"b066538d-4d14-4228-b563-bce796a28d0b","Type":"ContainerStarted","Data":"39f9e758e796c2c0a47624913a6194d955e4b7be579bb40a81695d84e8cc792d"}
Oct 30 21:47:32.822523 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:32.821999    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" event={"ID":"b066538d-4d14-4228-b563-bce796a28d0b","Type":"ContainerStarted","Data":"e31b0c31e0a5c99f58fcecf65b52e9e103f0f6730bbe396597a61489319d02f6"}
Oct 30 21:47:33.088215 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:33.088139    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:47:35.804731 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:35.804648    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:47:35.817574 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:35.817540    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:47:35.851953 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:35.851922    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:47:35.936282 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:35.936210    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" podStartSLOduration=4.936194144 podStartE2EDuration="4.936194144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:32.946699134 +0000 UTC m=+847.421203114" watchObservedRunningTime="2026-10-30 21:47:35.936194144 +0000 UTC m=+850.410698126"
Oct 30 21:47:42.088373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:42.088313    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:47:42.100773 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:42.100749    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:47:42.889032 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:42.889006    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:47:46.966558 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:46.966523    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"]
Oct 30 21:47:46.970633 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:46.970612    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:47:47.023160 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:47.023127    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-bc009f\"/\"openshift-service-ca.crt\""
Oct 30 21:47:47.072555 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:47.072521    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-bc009f\"/\"kube-root-ca.crt\""
Oct 30 21:47:47.072720 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:47.072603    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-bc009f\"/\"simulator-w3-bc009f-kserve-self-signed-certs\""
Oct 30 21:47:47.072720 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:47.072651    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-bc009f\"/\"default-dockercfg-hz59j\""
Oct 30 21:47:47.074832 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:47.074809    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"]
Oct 30 21:47:47.112105 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:47.112075    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqklm\" (UniqueName: \"kubernetes.io/projected/8606cde7-ffa5-48c3-941e-989ff637b916-kube-api-access-sqklm\") pod \"simulator-w3-bc009f-kserve-5d7667c858-q26tt\" (UID: \"8606cde7-ffa5-48c3-941e-989ff637b916\") " pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:47:47.112268 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:47.112123    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8606cde7-ffa5-48c3-941e-989ff637b916-home\") pod \"simulator-w3-bc009f-kserve-5d7667c858-q26tt\" (UID: \"8606cde7-ffa5-48c3-941e-989ff637b916\") " pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:47:47.112268 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:47.112235    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8606cde7-ffa5-48c3-941e-989ff637b916-dshm\") pod \"simulator-w3-bc009f-kserve-5d7667c858-q26tt\" (UID: \"8606cde7-ffa5-48c3-941e-989ff637b916\") " pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:47:47.112431 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:47.112275    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8606cde7-ffa5-48c3-941e-989ff637b916-tls-certs\") pod \"simulator-w3-bc009f-kserve-5d7667c858-q26tt\" (UID: \"8606cde7-ffa5-48c3-941e-989ff637b916\") " pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:47:47.112431 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:47.112330    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8606cde7-ffa5-48c3-941e-989ff637b916-tmp-dir\") pod \"simulator-w3-bc009f-kserve-5d7667c858-q26tt\" (UID: \"8606cde7-ffa5-48c3-941e-989ff637b916\") " pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:47:47.112531 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:47.112472    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8606cde7-ffa5-48c3-941e-989ff637b916-model-cache\") pod \"simulator-w3-bc009f-kserve-5d7667c858-q26tt\" (UID: \"8606cde7-ffa5-48c3-941e-989ff637b916\") " pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:47:47.168279 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:47:47.168244    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:47:47.213083 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:47.213043    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8606cde7-ffa5-48c3-941e-989ff637b916-model-cache\") pod \"simulator-w3-bc009f-kserve-5d7667c858-q26tt\" (UID: \"8606cde7-ffa5-48c3-941e-989ff637b916\") " pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:47:47.213272 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:47.213127    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqklm\" (UniqueName: \"kubernetes.io/projected/8606cde7-ffa5-48c3-941e-989ff637b916-kube-api-access-sqklm\") pod \"simulator-w3-bc009f-kserve-5d7667c858-q26tt\" (UID: \"8606cde7-ffa5-48c3-941e-989ff637b916\") " pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:47:47.213272 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:47.213165    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8606cde7-ffa5-48c3-941e-989ff637b916-home\") pod \"simulator-w3-bc009f-kserve-5d7667c858-q26tt\" (UID: \"8606cde7-ffa5-48c3-941e-989ff637b916\") " pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:47:47.213272 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:47.213212    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8606cde7-ffa5-48c3-941e-989ff637b916-dshm\") pod \"simulator-w3-bc009f-kserve-5d7667c858-q26tt\" (UID: \"8606cde7-ffa5-48c3-941e-989ff637b916\") " pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:47:47.213272 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:47.213239    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8606cde7-ffa5-48c3-941e-989ff637b916-tls-certs\") pod \"simulator-w3-bc009f-kserve-5d7667c858-q26tt\" (UID: \"8606cde7-ffa5-48c3-941e-989ff637b916\") " pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:47:47.213527 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:47.213284    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8606cde7-ffa5-48c3-941e-989ff637b916-tmp-dir\") pod \"simulator-w3-bc009f-kserve-5d7667c858-q26tt\" (UID: \"8606cde7-ffa5-48c3-941e-989ff637b916\") " pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:47:47.213579 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:47.213537    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8606cde7-ffa5-48c3-941e-989ff637b916-model-cache\") pod \"simulator-w3-bc009f-kserve-5d7667c858-q26tt\" (UID: \"8606cde7-ffa5-48c3-941e-989ff637b916\") " pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:47:47.213661 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:47.213630    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8606cde7-ffa5-48c3-941e-989ff637b916-home\") pod \"simulator-w3-bc009f-kserve-5d7667c858-q26tt\" (UID: \"8606cde7-ffa5-48c3-941e-989ff637b916\") " pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:47:47.213741 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:47.213725    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8606cde7-ffa5-48c3-941e-989ff637b916-tmp-dir\") pod \"simulator-w3-bc009f-kserve-5d7667c858-q26tt\" (UID: \"8606cde7-ffa5-48c3-941e-989ff637b916\") " pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:47:47.215950 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:47.215916    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8606cde7-ffa5-48c3-941e-989ff637b916-dshm\") pod \"simulator-w3-bc009f-kserve-5d7667c858-q26tt\" (UID: \"8606cde7-ffa5-48c3-941e-989ff637b916\") " pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:47:47.216165 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:47.216146    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8606cde7-ffa5-48c3-941e-989ff637b916-tls-certs\") pod \"simulator-w3-bc009f-kserve-5d7667c858-q26tt\" (UID: \"8606cde7-ffa5-48c3-941e-989ff637b916\") " pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:47:47.280700 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:47.280666    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqklm\" (UniqueName: \"kubernetes.io/projected/8606cde7-ffa5-48c3-941e-989ff637b916-kube-api-access-sqklm\") pod \"simulator-w3-bc009f-kserve-5d7667c858-q26tt\" (UID: \"8606cde7-ffa5-48c3-941e-989ff637b916\") " pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:47:47.283528 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:47.283503    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:47:47.686479 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:47.686454    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"]
Oct 30 21:47:47.688663 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:47:47.688629    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8606cde7_ffa5_48c3_941e_989ff637b916.slice/crio-11ea41fa6a29854dcdf6215e01dc3ca5bef9469918a45cd5b6086ee65b4e347a WatchSource:0}: Error finding container 11ea41fa6a29854dcdf6215e01dc3ca5bef9469918a45cd5b6086ee65b4e347a: Status 404 returned error can't find the container with id 11ea41fa6a29854dcdf6215e01dc3ca5bef9469918a45cd5b6086ee65b4e347a
Oct 30 21:47:47.904530 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:47.904491    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" event={"ID":"8606cde7-ffa5-48c3-941e-989ff637b916","Type":"ContainerStarted","Data":"4ed24b27a2be1f12193774bb8c54ef2f9d5220c094b1436ecc99ad4e6025166a"}
Oct 30 21:47:47.904530 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:47.904529    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" event={"ID":"8606cde7-ffa5-48c3-941e-989ff637b916","Type":"ContainerStarted","Data":"11ea41fa6a29854dcdf6215e01dc3ca5bef9469918a45cd5b6086ee65b4e347a"}
Oct 30 21:47:48.023055 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:48.023003    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podStartSLOduration=2.022990779 podStartE2EDuration="2.022990779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:48.021913918 +0000 UTC m=+862.496417910" watchObservedRunningTime="2026-10-30 21:47:48.022990779 +0000 UTC m=+862.497494757"
Oct 30 21:47:48.283928 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:48.283854    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:47:57.284430 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:57.284388    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:47:57.297268 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:57.297242    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:47:57.974168 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:47:57.974137    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:48:05.726765 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:05.726729    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"]
Oct 30 21:48:05.731446 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:05.731423    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:48:05.785672 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:05.785643    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-9b51e1\"/\"default-dockercfg-kt4rt\""
Oct 30 21:48:05.785672 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:05.785649    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-9b51e1\"/\"kube-root-ca.crt\""
Oct 30 21:48:05.796174 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:05.796148    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a9f73b-bb1f-4270-a38d-3703c46fae07-tls-certs\") pod \"simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc\" (UID: \"f6a9f73b-bb1f-4270-a38d-3703c46fae07\") " pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:48:05.796324 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:05.796199    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a9f73b-bb1f-4270-a38d-3703c46fae07-tmp-dir\") pod \"simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc\" (UID: \"f6a9f73b-bb1f-4270-a38d-3703c46fae07\") " pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:48:05.796324 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:05.796294    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfksp\" (UniqueName: \"kubernetes.io/projected/f6a9f73b-bb1f-4270-a38d-3703c46fae07-kube-api-access-gfksp\") pod \"simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc\" (UID: \"f6a9f73b-bb1f-4270-a38d-3703c46fae07\") " pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:48:05.796491 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:05.796439    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a9f73b-bb1f-4270-a38d-3703c46fae07-model-cache\") pod \"simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc\" (UID: \"f6a9f73b-bb1f-4270-a38d-3703c46fae07\") " pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:48:05.796543 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:05.796498    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a9f73b-bb1f-4270-a38d-3703c46fae07-home\") pod \"simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc\" (UID: \"f6a9f73b-bb1f-4270-a38d-3703c46fae07\") " pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:48:05.796620 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:05.796590    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a9f73b-bb1f-4270-a38d-3703c46fae07-dshm\") pod \"simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc\" (UID: \"f6a9f73b-bb1f-4270-a38d-3703c46fae07\") " pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:48:05.835099 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:05.835071    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-9b51e1\"/\"openshift-service-ca.crt\""
Oct 30 21:48:05.837448 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:05.837410    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"]
Oct 30 21:48:05.892984 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:05.892956    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-9b51e1\"/\"simulator-w5-9b51e1-kserve-self-signed-certs\""
Oct 30 21:48:05.897539 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:05.897512    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a9f73b-bb1f-4270-a38d-3703c46fae07-model-cache\") pod \"simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc\" (UID: \"f6a9f73b-bb1f-4270-a38d-3703c46fae07\") " pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:48:05.897680 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:05.897573    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a9f73b-bb1f-4270-a38d-3703c46fae07-home\") pod \"simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc\" (UID: \"f6a9f73b-bb1f-4270-a38d-3703c46fae07\") " pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:48:05.897680 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:05.897618    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a9f73b-bb1f-4270-a38d-3703c46fae07-dshm\") pod \"simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc\" (UID: \"f6a9f73b-bb1f-4270-a38d-3703c46fae07\") " pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:48:05.897680 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:05.897671    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a9f73b-bb1f-4270-a38d-3703c46fae07-tls-certs\") pod \"simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc\" (UID: \"f6a9f73b-bb1f-4270-a38d-3703c46fae07\") " pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:48:05.897896 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:05.897729    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a9f73b-bb1f-4270-a38d-3703c46fae07-tmp-dir\") pod \"simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc\" (UID: \"f6a9f73b-bb1f-4270-a38d-3703c46fae07\") " pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:48:05.897896 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:05.897791    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfksp\" (UniqueName: \"kubernetes.io/projected/f6a9f73b-bb1f-4270-a38d-3703c46fae07-kube-api-access-gfksp\") pod \"simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc\" (UID: \"f6a9f73b-bb1f-4270-a38d-3703c46fae07\") " pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:48:05.897992 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:05.897965    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a9f73b-bb1f-4270-a38d-3703c46fae07-model-cache\") pod \"simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc\" (UID: \"f6a9f73b-bb1f-4270-a38d-3703c46fae07\") " pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:48:05.898043 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:05.898019    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a9f73b-bb1f-4270-a38d-3703c46fae07-home\") pod \"simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc\" (UID: \"f6a9f73b-bb1f-4270-a38d-3703c46fae07\") " pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:48:05.898110 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:05.898094    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a9f73b-bb1f-4270-a38d-3703c46fae07-tmp-dir\") pod \"simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc\" (UID: \"f6a9f73b-bb1f-4270-a38d-3703c46fae07\") " pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:48:05.900446 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:05.900421    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a9f73b-bb1f-4270-a38d-3703c46fae07-dshm\") pod \"simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc\" (UID: \"f6a9f73b-bb1f-4270-a38d-3703c46fae07\") " pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:48:05.900631 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:05.900610    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a9f73b-bb1f-4270-a38d-3703c46fae07-tls-certs\") pod \"simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc\" (UID: \"f6a9f73b-bb1f-4270-a38d-3703c46fae07\") " pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:48:05.957019 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:05.956986    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfksp\" (UniqueName: \"kubernetes.io/projected/f6a9f73b-bb1f-4270-a38d-3703c46fae07-kube-api-access-gfksp\") pod \"simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc\" (UID: \"f6a9f73b-bb1f-4270-a38d-3703c46fae07\") " pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:48:06.044188 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:06.044158    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:48:06.433743 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:06.433713    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"]
Oct 30 21:48:06.435125 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:48:06.435093    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6a9f73b_bb1f_4270_a38d_3703c46fae07.slice/crio-1d26303041bb547d1b3a13a4d85d6478d42f6d7a06311c902357bda5b6660fd4 WatchSource:0}: Error finding container 1d26303041bb547d1b3a13a4d85d6478d42f6d7a06311c902357bda5b6660fd4: Status 404 returned error can't find the container with id 1d26303041bb547d1b3a13a4d85d6478d42f6d7a06311c902357bda5b6660fd4
Oct 30 21:48:07.015129 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:07.015087    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc" event={"ID":"f6a9f73b-bb1f-4270-a38d-3703c46fae07","Type":"ContainerStarted","Data":"ce3a1be70a92bba361c420e743a45b06804c17f31f1eb76ea630c8fd68f74b9c"}
Oct 30 21:48:07.015129 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:07.015134    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc" event={"ID":"f6a9f73b-bb1f-4270-a38d-3703c46fae07","Type":"ContainerStarted","Data":"1d26303041bb547d1b3a13a4d85d6478d42f6d7a06311c902357bda5b6660fd4"}
Oct 30 21:48:07.044484 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:07.044444    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:48:07.158959 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:07.158895    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc" podStartSLOduration=2.158878783 podStartE2EDuration="2.158878783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:07.156763998 +0000 UTC m=+881.631267973" watchObservedRunningTime="2026-10-30 21:48:07.158878783 +0000 UTC m=+881.633382765"
Oct 30 21:48:16.044279 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:16.044251    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:48:16.058973 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:16.058941    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:48:16.071686 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:16.071666    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:48:17.166628 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:48:17.166597    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:48:25.804521 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:25.804491    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 21:48:25.807871 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:25.807846    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 21:48:25.818969 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:25.818943    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 21:48:25.821256 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:25.821238    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 21:48:47.167299 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:48:47.167260    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:48:57.748218 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:57.748185    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"]
Oct 30 21:48:57.752717 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:57.752698    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:48:57.844713 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:57.844679    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cab3f652-e0fa-4351-b94a-271180e6c277-dshm\") pod \"premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf\" (UID: \"cab3f652-e0fa-4351-b94a-271180e6c277\") " pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:48:57.844978 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:57.844945    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8prf\" (UniqueName: \"kubernetes.io/projected/cab3f652-e0fa-4351-b94a-271180e6c277-kube-api-access-w8prf\") pod \"premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf\" (UID: \"cab3f652-e0fa-4351-b94a-271180e6c277\") " pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:48:57.845112 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:57.845002    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cab3f652-e0fa-4351-b94a-271180e6c277-home\") pod \"premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf\" (UID: \"cab3f652-e0fa-4351-b94a-271180e6c277\") " pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:48:57.845112 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:57.845033    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cab3f652-e0fa-4351-b94a-271180e6c277-tls-certs\") pod \"premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf\" (UID: \"cab3f652-e0fa-4351-b94a-271180e6c277\") " pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:48:57.845112 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:57.845102    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cab3f652-e0fa-4351-b94a-271180e6c277-tmp-dir\") pod \"premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf\" (UID: \"cab3f652-e0fa-4351-b94a-271180e6c277\") " pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:48:57.845288 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:57.845219    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cab3f652-e0fa-4351-b94a-271180e6c277-model-cache\") pod \"premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf\" (UID: \"cab3f652-e0fa-4351-b94a-271180e6c277\") " pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:48:57.854464 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:57.854438    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-c3e2d8\"/\"premium-w4-c3e2d8-kserve-self-signed-certs\""
Oct 30 21:48:57.856669 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:57.856650    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"]
Oct 30 21:48:57.946714 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:57.946661    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cab3f652-e0fa-4351-b94a-271180e6c277-tmp-dir\") pod \"premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf\" (UID: \"cab3f652-e0fa-4351-b94a-271180e6c277\") " pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:48:57.946906 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:57.946790    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cab3f652-e0fa-4351-b94a-271180e6c277-model-cache\") pod \"premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf\" (UID: \"cab3f652-e0fa-4351-b94a-271180e6c277\") " pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:48:57.947256 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:57.946963    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cab3f652-e0fa-4351-b94a-271180e6c277-dshm\") pod \"premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf\" (UID: \"cab3f652-e0fa-4351-b94a-271180e6c277\") " pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:48:57.947256 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:57.946984    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8prf\" (UniqueName: \"kubernetes.io/projected/cab3f652-e0fa-4351-b94a-271180e6c277-kube-api-access-w8prf\") pod \"premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf\" (UID: \"cab3f652-e0fa-4351-b94a-271180e6c277\") " pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:48:57.947256 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:57.947107    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cab3f652-e0fa-4351-b94a-271180e6c277-home\") pod \"premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf\" (UID: \"cab3f652-e0fa-4351-b94a-271180e6c277\") " pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:48:57.947256 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:57.947139    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cab3f652-e0fa-4351-b94a-271180e6c277-tls-certs\") pod \"premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf\" (UID: \"cab3f652-e0fa-4351-b94a-271180e6c277\") " pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:48:57.948152 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:57.948128    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cab3f652-e0fa-4351-b94a-271180e6c277-model-cache\") pod \"premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf\" (UID: \"cab3f652-e0fa-4351-b94a-271180e6c277\") " pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:48:57.948272 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:57.948242    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cab3f652-e0fa-4351-b94a-271180e6c277-tmp-dir\") pod \"premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf\" (UID: \"cab3f652-e0fa-4351-b94a-271180e6c277\") " pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:48:57.948415 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:57.948341    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cab3f652-e0fa-4351-b94a-271180e6c277-home\") pod \"premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf\" (UID: \"cab3f652-e0fa-4351-b94a-271180e6c277\") " pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:48:57.951396 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:57.950614    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cab3f652-e0fa-4351-b94a-271180e6c277-dshm\") pod \"premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf\" (UID: \"cab3f652-e0fa-4351-b94a-271180e6c277\") " pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:48:57.951396 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:57.950900    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cab3f652-e0fa-4351-b94a-271180e6c277-tls-certs\") pod \"premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf\" (UID: \"cab3f652-e0fa-4351-b94a-271180e6c277\") " pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:48:58.005000 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:58.004923    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8prf\" (UniqueName: \"kubernetes.io/projected/cab3f652-e0fa-4351-b94a-271180e6c277-kube-api-access-w8prf\") pod \"premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf\" (UID: \"cab3f652-e0fa-4351-b94a-271180e6c277\") " pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:48:58.064795 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:58.064764    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:48:58.466299 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:58.466276    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"]
Oct 30 21:48:58.468482 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:48:58.468459    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcab3f652_e0fa_4351_b94a_271180e6c277.slice/crio-412bcfae82b9355383bbb99e81c4650069ecdb7da4ab6bcae217ede471edb07f WatchSource:0}: Error finding container 412bcfae82b9355383bbb99e81c4650069ecdb7da4ab6bcae217ede471edb07f: Status 404 returned error can't find the container with id 412bcfae82b9355383bbb99e81c4650069ecdb7da4ab6bcae217ede471edb07f
Oct 30 21:48:59.303006 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:59.302965    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" event={"ID":"cab3f652-e0fa-4351-b94a-271180e6c277","Type":"ContainerStarted","Data":"0b466c23455d3ba73bcb706f91b4ad6a5d0cb4c8f745256b45d4967a40e3e64f"}
Oct 30 21:48:59.303400 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:59.303012    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" event={"ID":"cab3f652-e0fa-4351-b94a-271180e6c277","Type":"ContainerStarted","Data":"412bcfae82b9355383bbb99e81c4650069ecdb7da4ab6bcae217ede471edb07f"}
Oct 30 21:48:59.419120 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:48:59.419067    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podStartSLOduration=2.419054402 podStartE2EDuration="2.419054402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:48:59.417414105 +0000 UTC m=+933.891918081" watchObservedRunningTime="2026-10-30 21:48:59.419054402 +0000 UTC m=+933.893558381"
Oct 30 21:49:00.065065 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:00.065035    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:49:00.139531 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:00.139501    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad56773-dl4px"]
Oct 30 21:49:00.143449 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:00.143427    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad56773-dl4px"
Oct 30 21:49:00.171226 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:00.171200    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be908977-826e-4686-a7bf-035a233afac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bad56773-dl4px\" (UID: \"be908977-826e-4686-a7bf-035a233afac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad56773-dl4px"
Oct 30 21:49:00.171416 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:00.171397    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be908977-826e-4686-a7bf-035a233afac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bad56773-dl4px\" (UID: \"be908977-826e-4686-a7bf-035a233afac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad56773-dl4px"
Oct 30 21:49:00.171489 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:00.171440    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmdt5\" (UniqueName: \"kubernetes.io/projected/be908977-826e-4686-a7bf-035a233afac5-kube-api-access-zmdt5\") pod \"maas-api-revoke-sub-bad56773-dl4px\" (UID: \"be908977-826e-4686-a7bf-035a233afac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad56773-dl4px"
Oct 30 21:49:00.198457 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:00.198433    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:49:00.250494 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:00.250460    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad56773-dl4px"]
Oct 30 21:49:00.251577 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:00.251556    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:49:00.273100 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:00.273065    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmdt5\" (UniqueName: \"kubernetes.io/projected/be908977-826e-4686-a7bf-035a233afac5-kube-api-access-zmdt5\") pod \"maas-api-revoke-sub-bad56773-dl4px\" (UID: \"be908977-826e-4686-a7bf-035a233afac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad56773-dl4px"
Oct 30 21:49:00.273249 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:00.273151    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be908977-826e-4686-a7bf-035a233afac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bad56773-dl4px\" (UID: \"be908977-826e-4686-a7bf-035a233afac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad56773-dl4px"
Oct 30 21:49:00.273339 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:00.273320    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be908977-826e-4686-a7bf-035a233afac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bad56773-dl4px\" (UID: \"be908977-826e-4686-a7bf-035a233afac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad56773-dl4px"
Oct 30 21:49:00.274245 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:00.274220    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be908977-826e-4686-a7bf-035a233afac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bad56773-dl4px\" (UID: \"be908977-826e-4686-a7bf-035a233afac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad56773-dl4px"
Oct 30 21:49:00.276018 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:00.275997    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be908977-826e-4686-a7bf-035a233afac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bad56773-dl4px\" (UID: \"be908977-826e-4686-a7bf-035a233afac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad56773-dl4px"
Oct 30 21:49:00.338205 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:00.338133    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmdt5\" (UniqueName: \"kubernetes.io/projected/be908977-826e-4686-a7bf-035a233afac5-kube-api-access-zmdt5\") pod \"maas-api-revoke-sub-bad56773-dl4px\" (UID: \"be908977-826e-4686-a7bf-035a233afac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad56773-dl4px"
Oct 30 21:49:00.457367 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:00.457315    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad56773-dl4px"
Oct 30 21:49:00.830476 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:00.830407    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad56773-dl4px"]
Oct 30 21:49:00.832888 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:49:00.832857    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe908977_826e_4686_a7bf_035a233afac5.slice/crio-5e71deae5071934ef1b3b78a2495e23431faf5d6c71e5941abdb385eaa475231 WatchSource:0}: Error finding container 5e71deae5071934ef1b3b78a2495e23431faf5d6c71e5941abdb385eaa475231: Status 404 returned error can't find the container with id 5e71deae5071934ef1b3b78a2495e23431faf5d6c71e5941abdb385eaa475231
Oct 30 21:49:01.319597 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:01.319563    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad56773-dl4px" event={"ID":"be908977-826e-4686-a7bf-035a233afac5","Type":"ContainerStarted","Data":"5e71deae5071934ef1b3b78a2495e23431faf5d6c71e5941abdb385eaa475231"}
Oct 30 21:49:05.345805 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:05.345763    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad56773-dl4px" event={"ID":"be908977-826e-4686-a7bf-035a233afac5","Type":"ContainerStarted","Data":"cc5f22100064da7e6408b232086cd285fc5b4a3ccdf79a3506f08f329dab556b"}
Oct 30 21:49:05.459993 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:05.459930    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad56773-dl4px" podStartSLOduration=1.744421045 podStartE2EDuration="5.459910489s" totalImagesPullingTime="3.715489444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:05.458138309 +0000 UTC m=+939.932642290" watchObservedRunningTime="2026-10-30 21:49:05.459910489 +0000 UTC m=+939.934414468"
Oct 30 21:49:06.354847 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:06.354809    5087 generic.go:356] "Generic (PLEG): container finished" podID="be908977-826e-4686-a7bf-035a233afac5" containerID="cc5f22100064da7e6408b232086cd285fc5b4a3ccdf79a3506f08f329dab556b" exitCode=0
Oct 30 21:49:06.355288 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:06.354906    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad56773-dl4px" event={"ID":"be908977-826e-4686-a7bf-035a233afac5","Type":"ContainerDied","Data":"cc5f22100064da7e6408b232086cd285fc5b4a3ccdf79a3506f08f329dab556b"}
Oct 30 21:49:07.518413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:07.518311    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad56773-dl4px"
Oct 30 21:49:07.654423 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:07.652682    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be908977-826e-4686-a7bf-035a233afac5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be908977-826e-4686-a7bf-035a233afac5-maas-api-openshift-ca\") pod \"be908977-826e-4686-a7bf-035a233afac5\" (UID: \"be908977-826e-4686-a7bf-035a233afac5\") "
Oct 30 21:49:07.654423 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:07.652729    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be908977-826e-4686-a7bf-035a233afac5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be908977-826e-4686-a7bf-035a233afac5-maas-api-xks-ca\") pod \"be908977-826e-4686-a7bf-035a233afac5\" (UID: \"be908977-826e-4686-a7bf-035a233afac5\") "
Oct 30 21:49:07.654423 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:07.652789    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be908977-826e-4686-a7bf-035a233afac5-kube-api-access-zmdt5\" (UniqueName: \"kubernetes.io/projected/be908977-826e-4686-a7bf-035a233afac5-kube-api-access-zmdt5\") pod \"be908977-826e-4686-a7bf-035a233afac5\" (UID: \"be908977-826e-4686-a7bf-035a233afac5\") "
Oct 30 21:49:07.663181 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:07.663148    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be908977-826e-4686-a7bf-035a233afac5-maas-api-openshift-ca" pod "be908977-826e-4686-a7bf-035a233afac5" (UID: "be908977-826e-4686-a7bf-035a233afac5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:49:07.663933 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:07.663907    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be908977-826e-4686-a7bf-035a233afac5-kube-api-access-zmdt5" pod "be908977-826e-4686-a7bf-035a233afac5" (UID: "be908977-826e-4686-a7bf-035a233afac5"). InnerVolumeSpecName "kube-api-access-zmdt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:07.670802 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:07.670774    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be908977-826e-4686-a7bf-035a233afac5-maas-api-xks-ca" pod "be908977-826e-4686-a7bf-035a233afac5" (UID: "be908977-826e-4686-a7bf-035a233afac5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:49:07.754229 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:07.754200    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be908977-826e-4686-a7bf-035a233afac5-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:07.754229 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:07.754224    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be908977-826e-4686-a7bf-035a233afac5-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:07.754229 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:07.754235    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmdt5\" (UniqueName: \"kubernetes.io/projected/be908977-826e-4686-a7bf-035a233afac5-kube-api-access-zmdt5\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:08.065861 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.065830    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:49:08.081113 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.081088    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:49:08.367116 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.367034    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad56773-dl4px"
Oct 30 21:49:08.367116 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.367074    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bad56773-dl4px" event={"ID":"be908977-826e-4686-a7bf-035a233afac5","Type":"ContainerDied","Data":"5e71deae5071934ef1b3b78a2495e23431faf5d6c71e5941abdb385eaa475231"}
Oct 30 21:49:08.367380 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.367154    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e71deae5071934ef1b3b78a2495e23431faf5d6c71e5941abdb385eaa475231"
Oct 30 21:49:08.378992 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.378968    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:49:08.457496 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.457464    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"]
Oct 30 21:49:08.457961 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.457948    5087 cpu_manager.go:414] "containerMap: removing container" podUID="be908977-826e-4686-a7bf-035a233afac5" containerName="revoke-keys"
Oct 30 21:49:08.458014 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.457962    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be908977-826e-4686-a7bf-035a233afac5" containerName="revoke-keys"
Oct 30 21:49:08.458051 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.458042    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="be908977-826e-4686-a7bf-035a233afac5" containerName="revoke-keys"
Oct 30 21:49:08.461394 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.461375    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:49:08.516027 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.515986    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-bc009f\"/\"premium-w3-bc009f-kserve-self-signed-certs\""
Oct 30 21:49:08.518408 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.518329    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"]
Oct 30 21:49:08.562608 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.562578    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-tmp-dir\") pod \"premium-w3-bc009f-kserve-55c859665f-49gfw\" (UID: \"1ae4d8fa-1b3c-45dd-8461-edfcc9707043\") " pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:49:08.562996 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.562614    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-home\") pod \"premium-w3-bc009f-kserve-55c859665f-49gfw\" (UID: \"1ae4d8fa-1b3c-45dd-8461-edfcc9707043\") " pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:49:08.562996 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.562642    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-tls-certs\") pod \"premium-w3-bc009f-kserve-55c859665f-49gfw\" (UID: \"1ae4d8fa-1b3c-45dd-8461-edfcc9707043\") " pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:49:08.562996 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.562774    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-dshm\") pod \"premium-w3-bc009f-kserve-55c859665f-49gfw\" (UID: \"1ae4d8fa-1b3c-45dd-8461-edfcc9707043\") " pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:49:08.562996 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.562851    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76v2f\" (UniqueName: \"kubernetes.io/projected/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-kube-api-access-76v2f\") pod \"premium-w3-bc009f-kserve-55c859665f-49gfw\" (UID: \"1ae4d8fa-1b3c-45dd-8461-edfcc9707043\") " pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:49:08.562996 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.562960    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-model-cache\") pod \"premium-w3-bc009f-kserve-55c859665f-49gfw\" (UID: \"1ae4d8fa-1b3c-45dd-8461-edfcc9707043\") " pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:49:08.664069 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.663989    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-tmp-dir\") pod \"premium-w3-bc009f-kserve-55c859665f-49gfw\" (UID: \"1ae4d8fa-1b3c-45dd-8461-edfcc9707043\") " pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:49:08.664069 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.664023    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-home\") pod \"premium-w3-bc009f-kserve-55c859665f-49gfw\" (UID: \"1ae4d8fa-1b3c-45dd-8461-edfcc9707043\") " pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:49:08.664069 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.664057    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-tls-certs\") pod \"premium-w3-bc009f-kserve-55c859665f-49gfw\" (UID: \"1ae4d8fa-1b3c-45dd-8461-edfcc9707043\") " pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:49:08.664386 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.664089    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-dshm\") pod \"premium-w3-bc009f-kserve-55c859665f-49gfw\" (UID: \"1ae4d8fa-1b3c-45dd-8461-edfcc9707043\") " pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:49:08.664386 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.664134    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76v2f\" (UniqueName: \"kubernetes.io/projected/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-kube-api-access-76v2f\") pod \"premium-w3-bc009f-kserve-55c859665f-49gfw\" (UID: \"1ae4d8fa-1b3c-45dd-8461-edfcc9707043\") " pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:49:08.664386 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.664166    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-model-cache\") pod \"premium-w3-bc009f-kserve-55c859665f-49gfw\" (UID: \"1ae4d8fa-1b3c-45dd-8461-edfcc9707043\") " pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:49:08.664878 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.664478    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-home\") pod \"premium-w3-bc009f-kserve-55c859665f-49gfw\" (UID: \"1ae4d8fa-1b3c-45dd-8461-edfcc9707043\") " pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:49:08.664878 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.664534    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-tmp-dir\") pod \"premium-w3-bc009f-kserve-55c859665f-49gfw\" (UID: \"1ae4d8fa-1b3c-45dd-8461-edfcc9707043\") " pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:49:08.664878 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.664603    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-model-cache\") pod \"premium-w3-bc009f-kserve-55c859665f-49gfw\" (UID: \"1ae4d8fa-1b3c-45dd-8461-edfcc9707043\") " pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:49:08.667190 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.667166    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-dshm\") pod \"premium-w3-bc009f-kserve-55c859665f-49gfw\" (UID: \"1ae4d8fa-1b3c-45dd-8461-edfcc9707043\") " pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:49:08.667450 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.667434    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-tls-certs\") pod \"premium-w3-bc009f-kserve-55c859665f-49gfw\" (UID: \"1ae4d8fa-1b3c-45dd-8461-edfcc9707043\") " pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:49:08.721949 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.721919    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76v2f\" (UniqueName: \"kubernetes.io/projected/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-kube-api-access-76v2f\") pod \"premium-w3-bc009f-kserve-55c859665f-49gfw\" (UID: \"1ae4d8fa-1b3c-45dd-8461-edfcc9707043\") " pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:49:08.773316 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:08.773284    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:49:09.169842 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:09.169811    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"]
Oct 30 21:49:09.171493 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:49:09.171460    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ae4d8fa_1b3c_45dd_8461_edfcc9707043.slice/crio-445607c30fe69d33f7e6c5ab715d28138eaa12767a40288ac7cf5efa74e9d0ab WatchSource:0}: Error finding container 445607c30fe69d33f7e6c5ab715d28138eaa12767a40288ac7cf5efa74e9d0ab: Status 404 returned error can't find the container with id 445607c30fe69d33f7e6c5ab715d28138eaa12767a40288ac7cf5efa74e9d0ab
Oct 30 21:49:09.374677 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:09.374640    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" event={"ID":"1ae4d8fa-1b3c-45dd-8461-edfcc9707043","Type":"ContainerStarted","Data":"fe059f22e865360e5a4bc2ff2e119338c8a36721ccb5c36404a629f473c71da0"}
Oct 30 21:49:09.374677 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:09.374682    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" event={"ID":"1ae4d8fa-1b3c-45dd-8461-edfcc9707043","Type":"ContainerStarted","Data":"445607c30fe69d33f7e6c5ab715d28138eaa12767a40288ac7cf5efa74e9d0ab"}
Oct 30 21:49:09.490310 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:09.490260    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" podStartSLOduration=1.4902425 podStartE2EDuration="1.4902425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:09.489255494 +0000 UTC m=+943.963759475" watchObservedRunningTime="2026-10-30 21:49:09.4902425 +0000 UTC m=+943.964746479"
Oct 30 21:49:09.773551 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:09.773465    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:49:15.666018 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:15.665976    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53a6ea98-8qp8j"]
Oct 30 21:49:15.670079 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:15.670056    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a6ea98-8qp8j"
Oct 30 21:49:15.727743 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:15.727712    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53a6ea98-8qp8j"]
Oct 30 21:49:15.743472 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:15.743445    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ed21eb-f88e-4f82-a19b-179c3026d74a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53a6ea98-8qp8j\" (UID: \"c7ed21eb-f88e-4f82-a19b-179c3026d74a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a6ea98-8qp8j"
Oct 30 21:49:15.743659 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:15.743552    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gndzn\" (UniqueName: \"kubernetes.io/projected/c7ed21eb-f88e-4f82-a19b-179c3026d74a-kube-api-access-gndzn\") pod \"maas-api-revoke-sub-53a6ea98-8qp8j\" (UID: \"c7ed21eb-f88e-4f82-a19b-179c3026d74a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a6ea98-8qp8j"
Oct 30 21:49:15.743659 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:15.743588    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ed21eb-f88e-4f82-a19b-179c3026d74a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53a6ea98-8qp8j\" (UID: \"c7ed21eb-f88e-4f82-a19b-179c3026d74a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a6ea98-8qp8j"
Oct 30 21:49:15.774022 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:15.773991    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:49:15.774022 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:15.773996    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:49:15.844839 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:15.844803    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gndzn\" (UniqueName: \"kubernetes.io/projected/c7ed21eb-f88e-4f82-a19b-179c3026d74a-kube-api-access-gndzn\") pod \"maas-api-revoke-sub-53a6ea98-8qp8j\" (UID: \"c7ed21eb-f88e-4f82-a19b-179c3026d74a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a6ea98-8qp8j"
Oct 30 21:49:15.845038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:15.844873    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ed21eb-f88e-4f82-a19b-179c3026d74a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53a6ea98-8qp8j\" (UID: \"c7ed21eb-f88e-4f82-a19b-179c3026d74a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a6ea98-8qp8j"
Oct 30 21:49:15.845038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:15.844989    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ed21eb-f88e-4f82-a19b-179c3026d74a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53a6ea98-8qp8j\" (UID: \"c7ed21eb-f88e-4f82-a19b-179c3026d74a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a6ea98-8qp8j"
Oct 30 21:49:15.845641 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:15.845617    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ed21eb-f88e-4f82-a19b-179c3026d74a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53a6ea98-8qp8j\" (UID: \"c7ed21eb-f88e-4f82-a19b-179c3026d74a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a6ea98-8qp8j"
Oct 30 21:49:15.847502 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:15.847483    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ed21eb-f88e-4f82-a19b-179c3026d74a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53a6ea98-8qp8j\" (UID: \"c7ed21eb-f88e-4f82-a19b-179c3026d74a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a6ea98-8qp8j"
Oct 30 21:49:15.902528 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:15.902498    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gndzn\" (UniqueName: \"kubernetes.io/projected/c7ed21eb-f88e-4f82-a19b-179c3026d74a-kube-api-access-gndzn\") pod \"maas-api-revoke-sub-53a6ea98-8qp8j\" (UID: \"c7ed21eb-f88e-4f82-a19b-179c3026d74a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a6ea98-8qp8j"
Oct 30 21:49:15.987694 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:15.987663    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a6ea98-8qp8j"
Oct 30 21:49:16.365590 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:16.365564    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53a6ea98-8qp8j"]
Oct 30 21:49:16.368176 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:49:16.368097    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7ed21eb_f88e_4f82_a19b_179c3026d74a.slice/crio-83cbf2554d723cea30ca08f2ec3497885d7ff7345bdee4790d83e41ddbbf2778 WatchSource:0}: Error finding container 83cbf2554d723cea30ca08f2ec3497885d7ff7345bdee4790d83e41ddbbf2778: Status 404 returned error can't find the container with id 83cbf2554d723cea30ca08f2ec3497885d7ff7345bdee4790d83e41ddbbf2778
Oct 30 21:49:16.417250 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:16.417222    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a6ea98-8qp8j" event={"ID":"c7ed21eb-f88e-4f82-a19b-179c3026d74a","Type":"ContainerStarted","Data":"83cbf2554d723cea30ca08f2ec3497885d7ff7345bdee4790d83e41ddbbf2778"}
Oct 30 21:49:17.168995 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:49:17.168902    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:49:17.424809 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:17.424723    5087 generic.go:356] "Generic (PLEG): container finished" podID="c7ed21eb-f88e-4f82-a19b-179c3026d74a" containerID="b4be0fe9a1c26c8c5a78647c8cc349da3e905fb197d2733630dfadee01286828" exitCode=0
Oct 30 21:49:17.424953 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:17.424806    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a6ea98-8qp8j" event={"ID":"c7ed21eb-f88e-4f82-a19b-179c3026d74a","Type":"ContainerDied","Data":"b4be0fe9a1c26c8c5a78647c8cc349da3e905fb197d2733630dfadee01286828"}
Oct 30 21:49:18.588539 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:18.588516    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a6ea98-8qp8j"
Oct 30 21:49:18.679333 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:18.679297    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7ed21eb-f88e-4f82-a19b-179c3026d74a-kube-api-access-gndzn\" (UniqueName: \"kubernetes.io/projected/c7ed21eb-f88e-4f82-a19b-179c3026d74a-kube-api-access-gndzn\") pod \"c7ed21eb-f88e-4f82-a19b-179c3026d74a\" (UID: \"c7ed21eb-f88e-4f82-a19b-179c3026d74a\") "
Oct 30 21:49:18.679545 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:18.679447    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7ed21eb-f88e-4f82-a19b-179c3026d74a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ed21eb-f88e-4f82-a19b-179c3026d74a-maas-api-openshift-ca\") pod \"c7ed21eb-f88e-4f82-a19b-179c3026d74a\" (UID: \"c7ed21eb-f88e-4f82-a19b-179c3026d74a\") "
Oct 30 21:49:18.679615 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:18.679557    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7ed21eb-f88e-4f82-a19b-179c3026d74a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ed21eb-f88e-4f82-a19b-179c3026d74a-maas-api-xks-ca\") pod \"c7ed21eb-f88e-4f82-a19b-179c3026d74a\" (UID: \"c7ed21eb-f88e-4f82-a19b-179c3026d74a\") "
Oct 30 21:49:18.679840 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:18.679813    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7ed21eb-f88e-4f82-a19b-179c3026d74a-maas-api-openshift-ca" pod "c7ed21eb-f88e-4f82-a19b-179c3026d74a" (UID: "c7ed21eb-f88e-4f82-a19b-179c3026d74a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:49:18.680007 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:18.679947    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ed21eb-f88e-4f82-a19b-179c3026d74a-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:18.681586 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:18.681557    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ed21eb-f88e-4f82-a19b-179c3026d74a-kube-api-access-gndzn" pod "c7ed21eb-f88e-4f82-a19b-179c3026d74a" (UID: "c7ed21eb-f88e-4f82-a19b-179c3026d74a"). InnerVolumeSpecName "kube-api-access-gndzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:18.681828 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:18.681806    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7ed21eb-f88e-4f82-a19b-179c3026d74a-maas-api-xks-ca" pod "c7ed21eb-f88e-4f82-a19b-179c3026d74a" (UID: "c7ed21eb-f88e-4f82-a19b-179c3026d74a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:49:18.773763 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:18.773731    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:49:18.781116 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:18.781083    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ed21eb-f88e-4f82-a19b-179c3026d74a-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:18.781116 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:18.781116    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gndzn\" (UniqueName: \"kubernetes.io/projected/c7ed21eb-f88e-4f82-a19b-179c3026d74a-kube-api-access-gndzn\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:18.787065 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:18.787043    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:49:19.437475 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:19.437443    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a6ea98-8qp8j"
Oct 30 21:49:19.437652 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:19.437486    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53a6ea98-8qp8j" event={"ID":"c7ed21eb-f88e-4f82-a19b-179c3026d74a","Type":"ContainerDied","Data":"83cbf2554d723cea30ca08f2ec3497885d7ff7345bdee4790d83e41ddbbf2778"}
Oct 30 21:49:19.437652 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:19.437545    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83cbf2554d723cea30ca08f2ec3497885d7ff7345bdee4790d83e41ddbbf2778"
Oct 30 21:49:19.453039 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:19.453010    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:49:26.093750 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.093714    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"]
Oct 30 21:49:26.094231 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.094217    5087 cpu_manager.go:414] "containerMap: removing container" podUID="c7ed21eb-f88e-4f82-a19b-179c3026d74a" containerName="revoke-keys"
Oct 30 21:49:26.094271 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.094232    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7ed21eb-f88e-4f82-a19b-179c3026d74a" containerName="revoke-keys"
Oct 30 21:49:26.094317 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.094309    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7ed21eb-f88e-4f82-a19b-179c3026d74a" containerName="revoke-keys"
Oct 30 21:49:26.097718 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.097703    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:26.149732 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.149705    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-557708-openshift-default-dockercfg-87jdg\""
Oct 30 21:49:26.159445 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.159406    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-credential-socket\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:26.159588 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.159563    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-istio-data\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:26.159644 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.159606    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nr7t\" (UniqueName: \"kubernetes.io/projected/5337230d-c32c-42e6-8985-4ff157dd5b95-kube-api-access-6nr7t\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:26.159682 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.159641    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-workload-socket\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:26.159755 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.159735    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5337230d-c32c-42e6-8985-4ff157dd5b95-istio-podinfo\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:26.159850 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.159823    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5337230d-c32c-42e6-8985-4ff157dd5b95-istio-token\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:26.159975 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.159907    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-istio-envoy\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:26.160038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.159978    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5337230d-c32c-42e6-8985-4ff157dd5b95-istiod-ca-cert\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:26.160099 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.160032    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-workload-certs\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:26.201918 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.201888    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"]
Oct 30 21:49:26.260879 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.260842    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-istio-data\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:26.260879 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.260881    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nr7t\" (UniqueName: \"kubernetes.io/projected/5337230d-c32c-42e6-8985-4ff157dd5b95-kube-api-access-6nr7t\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:26.261113 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.260910    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-workload-socket\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:26.261113 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.260953    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5337230d-c32c-42e6-8985-4ff157dd5b95-istio-podinfo\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:26.261113 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.261003    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5337230d-c32c-42e6-8985-4ff157dd5b95-istio-token\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:26.261113 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.261043    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-istio-envoy\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:26.261113 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.261107    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5337230d-c32c-42e6-8985-4ff157dd5b95-istiod-ca-cert\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:26.261394 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.261157    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-workload-certs\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:26.261394 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.261190    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-credential-socket\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:26.261394 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.261308    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-istio-data\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:26.261394 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.261380    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-workload-socket\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:26.261615 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.261579    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-credential-socket\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:26.261765 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.261730    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-workload-certs\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:26.261915 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.261894    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5337230d-c32c-42e6-8985-4ff157dd5b95-istiod-ca-cert\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:26.264262 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.264244    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-istio-envoy\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:26.264614 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.264589    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5337230d-c32c-42e6-8985-4ff157dd5b95-istio-podinfo\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:26.319367 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.319313    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5337230d-c32c-42e6-8985-4ff157dd5b95-istio-token\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:26.319941 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.319920    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nr7t\" (UniqueName: \"kubernetes.io/projected/5337230d-c32c-42e6-8985-4ff157dd5b95-kube-api-access-6nr7t\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:26.414017 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.413929    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:26.850376 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.850325    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"]
Oct 30 21:49:26.850964 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:49:26.850934    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5337230d_c32c_42e6_8985_4ff157dd5b95.slice/crio-b14b5a8e259fb0d4fee48368aff3f1e16bf108fd61b3ecf01ecc3369ca53999f WatchSource:0}: Error finding container b14b5a8e259fb0d4fee48368aff3f1e16bf108fd61b3ecf01ecc3369ca53999f: Status 404 returned error can't find the container with id b14b5a8e259fb0d4fee48368aff3f1e16bf108fd61b3ecf01ecc3369ca53999f
Oct 30 21:49:26.853483 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.853450    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:49:26.853597 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.853517    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:49:26.853597 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:26.853547    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:49:27.490332 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:27.490295    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj" event={"ID":"5337230d-c32c-42e6-8985-4ff157dd5b95","Type":"ContainerStarted","Data":"8e4f25f698847b880a0306ff29c4e5aa651186c574f274b82ad529a2285e1952"}
Oct 30 21:49:27.490781 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:27.490368    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj" event={"ID":"5337230d-c32c-42e6-8985-4ff157dd5b95","Type":"ContainerStarted","Data":"b14b5a8e259fb0d4fee48368aff3f1e16bf108fd61b3ecf01ecc3369ca53999f"}
Oct 30 21:49:27.625630 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:27.625574    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj" podStartSLOduration=1.6255606729999998 podStartE2EDuration="1.625560673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:27.624522496 +0000 UTC m=+962.099026475" watchObservedRunningTime="2026-10-30 21:49:27.625560673 +0000 UTC m=+962.100064646"
Oct 30 21:49:28.414522 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:28.414492    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:28.416496 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:28.416463    5087 patch_prober.go:29] interesting pod/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 21:49:28.416637 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:28.416537    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj" podUID="5337230d-c32c-42e6-8985-4ff157dd5b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 21:49:28.697829 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:28.697733    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"]
Oct 30 21:49:28.698315 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:28.697971    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5" podUID="964d70b0-09d8-46d7-98fa-d034aa975096" containerName="istio-proxy" containerID="cri-o://3c32d0925d2cc3dd4b6e41e8c41b544a80ccecd94b07003d32319338d0174695" gracePeriod=30
Oct 30 21:49:28.755521 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:28.755483    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"]
Oct 30 21:49:28.755986 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:28.755952    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx" podUID="4928c741-8a88-4452-b2ce-f7ed6cff5304" containerName="istio-proxy" containerID="cri-o://4200d982d1b2efc9118d95eb28ec489523a5877766f8063891f88c208ed8be1a" gracePeriod=30
Oct 30 21:49:29.415229 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:29.415198    5087 patch_prober.go:29] interesting pod/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 21:49:29.415428 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:29.415254    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj" podUID="5337230d-c32c-42e6-8985-4ff157dd5b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 21:49:30.415086 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:30.415043    5087 patch_prober.go:29] interesting pod/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 21:49:30.415609 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:30.415107    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj" podUID="5337230d-c32c-42e6-8985-4ff157dd5b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 21:49:31.414692 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:31.414657    5087 patch_prober.go:29] interesting pod/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 21:49:31.414885 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:31.414722    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj" podUID="5337230d-c32c-42e6-8985-4ff157dd5b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 21:49:31.428866 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:31.428829    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5109e7be-q8vpg"]
Oct 30 21:49:31.433124 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:31.433103    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5109e7be-q8vpg"
Oct 30 21:49:31.487433 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:31.487398    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:49:31.520401 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:31.520375    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dptx2\" (UniqueName: \"kubernetes.io/projected/3ae089fd-bbb8-49d6-b323-0f8dac4b91d7-kube-api-access-dptx2\") pod \"maas-api-revoke-sub-5109e7be-q8vpg\" (UID: \"3ae089fd-bbb8-49d6-b323-0f8dac4b91d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5109e7be-q8vpg"
Oct 30 21:49:31.520566 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:31.520439    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ae089fd-bbb8-49d6-b323-0f8dac4b91d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5109e7be-q8vpg\" (UID: \"3ae089fd-bbb8-49d6-b323-0f8dac4b91d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5109e7be-q8vpg"
Oct 30 21:49:31.520606 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:31.520586    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ae089fd-bbb8-49d6-b323-0f8dac4b91d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5109e7be-q8vpg\" (UID: \"3ae089fd-bbb8-49d6-b323-0f8dac4b91d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5109e7be-q8vpg"
Oct 30 21:49:31.559371 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:31.551052    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:49:31.569808 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:31.569771    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5109e7be-q8vpg"]
Oct 30 21:49:31.622173 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:31.622138    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ae089fd-bbb8-49d6-b323-0f8dac4b91d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5109e7be-q8vpg\" (UID: \"3ae089fd-bbb8-49d6-b323-0f8dac4b91d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5109e7be-q8vpg"
Oct 30 21:49:31.622329 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:31.622226    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dptx2\" (UniqueName: \"kubernetes.io/projected/3ae089fd-bbb8-49d6-b323-0f8dac4b91d7-kube-api-access-dptx2\") pod \"maas-api-revoke-sub-5109e7be-q8vpg\" (UID: \"3ae089fd-bbb8-49d6-b323-0f8dac4b91d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5109e7be-q8vpg"
Oct 30 21:49:31.622329 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:31.622276    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ae089fd-bbb8-49d6-b323-0f8dac4b91d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5109e7be-q8vpg\" (UID: \"3ae089fd-bbb8-49d6-b323-0f8dac4b91d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5109e7be-q8vpg"
Oct 30 21:49:31.622819 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:31.622799    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ae089fd-bbb8-49d6-b323-0f8dac4b91d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5109e7be-q8vpg\" (UID: \"3ae089fd-bbb8-49d6-b323-0f8dac4b91d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5109e7be-q8vpg"
Oct 30 21:49:31.624740 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:31.624720    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ae089fd-bbb8-49d6-b323-0f8dac4b91d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5109e7be-q8vpg\" (UID: \"3ae089fd-bbb8-49d6-b323-0f8dac4b91d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5109e7be-q8vpg"
Oct 30 21:49:31.680069 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:31.679989    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dptx2\" (UniqueName: \"kubernetes.io/projected/3ae089fd-bbb8-49d6-b323-0f8dac4b91d7-kube-api-access-dptx2\") pod \"maas-api-revoke-sub-5109e7be-q8vpg\" (UID: \"3ae089fd-bbb8-49d6-b323-0f8dac4b91d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5109e7be-q8vpg"
Oct 30 21:49:31.773081 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:31.773047    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5109e7be-q8vpg"
Oct 30 21:49:32.159371 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:32.154785    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5109e7be-q8vpg"]
Oct 30 21:49:32.171390 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:49:32.171327    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae089fd_bbb8_49d6_b323_0f8dac4b91d7.slice/crio-1933ab13f86ca5f91ee4c9524df8abb203a6008cecf3944a1e6c841e7dbbf371 WatchSource:0}: Error finding container 1933ab13f86ca5f91ee4c9524df8abb203a6008cecf3944a1e6c841e7dbbf371: Status 404 returned error can't find the container with id 1933ab13f86ca5f91ee4c9524df8abb203a6008cecf3944a1e6c841e7dbbf371
Oct 30 21:49:32.415095 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:32.414971    5087 patch_prober.go:29] interesting pod/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body=
Oct 30 21:49:32.415095 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:32.415021    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj" podUID="5337230d-c32c-42e6-8985-4ff157dd5b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused"
Oct 30 21:49:32.522713 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:32.522679    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5109e7be-q8vpg" event={"ID":"3ae089fd-bbb8-49d6-b323-0f8dac4b91d7","Type":"ContainerStarted","Data":"1933ab13f86ca5f91ee4c9524df8abb203a6008cecf3944a1e6c841e7dbbf371"}
Oct 30 21:49:33.530995 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:33.530952    5087 generic.go:356] "Generic (PLEG): container finished" podID="3ae089fd-bbb8-49d6-b323-0f8dac4b91d7" containerID="cb3e2d9f0d01640756d6b5d7ffcd1e0ab0fc142b3f33fd45722206de673d986f" exitCode=0
Oct 30 21:49:33.531515 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:33.531051    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5109e7be-q8vpg" event={"ID":"3ae089fd-bbb8-49d6-b323-0f8dac4b91d7","Type":"ContainerDied","Data":"cb3e2d9f0d01640756d6b5d7ffcd1e0ab0fc142b3f33fd45722206de673d986f"}
Oct 30 21:49:33.603199 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:33.603169    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:33.603533 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:33.603513    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:33.604653 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:33.604630    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 21:49:34.051869 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.051450    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:49:34.074408 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.074383    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:49:34.149752 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.149708    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/964d70b0-09d8-46d7-98fa-d034aa975096-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/964d70b0-09d8-46d7-98fa-d034aa975096-istio-podinfo\") pod \"964d70b0-09d8-46d7-98fa-d034aa975096\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") "
Oct 30 21:49:34.149947 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.149783    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/964d70b0-09d8-46d7-98fa-d034aa975096-istio-token\" (UniqueName: \"kubernetes.io/projected/964d70b0-09d8-46d7-98fa-d034aa975096-istio-token\") pod \"964d70b0-09d8-46d7-98fa-d034aa975096\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") "
Oct 30 21:49:34.149947 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.149814    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-credential-socket\") pod \"964d70b0-09d8-46d7-98fa-d034aa975096\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") "
Oct 30 21:49:34.149947 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.149846    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-istio-data\") pod \"4928c741-8a88-4452-b2ce-f7ed6cff5304\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") "
Oct 30 21:49:34.149947 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.149875    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-workload-socket\") pod \"4928c741-8a88-4452-b2ce-f7ed6cff5304\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") "
Oct 30 21:49:34.149947 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.149926    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4928c741-8a88-4452-b2ce-f7ed6cff5304-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4928c741-8a88-4452-b2ce-f7ed6cff5304-istio-podinfo\") pod \"4928c741-8a88-4452-b2ce-f7ed6cff5304\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") "
Oct 30 21:49:34.150229 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.149953    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-istio-data\") pod \"964d70b0-09d8-46d7-98fa-d034aa975096\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") "
Oct 30 21:49:34.150229 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.149982    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-workload-socket\") pod \"964d70b0-09d8-46d7-98fa-d034aa975096\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") "
Oct 30 21:49:34.150229 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.150014    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/964d70b0-09d8-46d7-98fa-d034aa975096-kube-api-access-27pct\" (UniqueName: \"kubernetes.io/projected/964d70b0-09d8-46d7-98fa-d034aa975096-kube-api-access-27pct\") pod \"964d70b0-09d8-46d7-98fa-d034aa975096\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") "
Oct 30 21:49:34.150229 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.150054    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-workload-certs\") pod \"964d70b0-09d8-46d7-98fa-d034aa975096\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") "
Oct 30 21:49:34.150229 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.150097    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-credential-socket\") pod \"4928c741-8a88-4452-b2ce-f7ed6cff5304\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") "
Oct 30 21:49:34.150229 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.150145    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-istio-envoy\") pod \"4928c741-8a88-4452-b2ce-f7ed6cff5304\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") "
Oct 30 21:49:34.150229 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.150199    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-workload-certs\") pod \"4928c741-8a88-4452-b2ce-f7ed6cff5304\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") "
Oct 30 21:49:34.150640 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.150261    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4928c741-8a88-4452-b2ce-f7ed6cff5304-istio-token\" (UniqueName: \"kubernetes.io/projected/4928c741-8a88-4452-b2ce-f7ed6cff5304-istio-token\") pod \"4928c741-8a88-4452-b2ce-f7ed6cff5304\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") "
Oct 30 21:49:34.150640 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.150308    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-istio-envoy\") pod \"964d70b0-09d8-46d7-98fa-d034aa975096\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") "
Oct 30 21:49:34.150640 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.150336    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4928c741-8a88-4452-b2ce-f7ed6cff5304-kube-api-access-52wxm\" (UniqueName: \"kubernetes.io/projected/4928c741-8a88-4452-b2ce-f7ed6cff5304-kube-api-access-52wxm\") pod \"4928c741-8a88-4452-b2ce-f7ed6cff5304\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") "
Oct 30 21:49:34.150640 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.150418    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/964d70b0-09d8-46d7-98fa-d034aa975096-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/964d70b0-09d8-46d7-98fa-d034aa975096-istiod-ca-cert\") pod \"964d70b0-09d8-46d7-98fa-d034aa975096\" (UID: \"964d70b0-09d8-46d7-98fa-d034aa975096\") "
Oct 30 21:49:34.150640 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.150460    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4928c741-8a88-4452-b2ce-f7ed6cff5304-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4928c741-8a88-4452-b2ce-f7ed6cff5304-istiod-ca-cert\") pod \"4928c741-8a88-4452-b2ce-f7ed6cff5304\" (UID: \"4928c741-8a88-4452-b2ce-f7ed6cff5304\") "
Oct 30 21:49:34.150899 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.150669    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-workload-socket" pod "964d70b0-09d8-46d7-98fa-d034aa975096" (UID: "964d70b0-09d8-46d7-98fa-d034aa975096"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:34.150899 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.150822    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-istio-data" pod "964d70b0-09d8-46d7-98fa-d034aa975096" (UID: "964d70b0-09d8-46d7-98fa-d034aa975096"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:34.151010 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.150921    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:34.151010 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.150942    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:34.151135 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.151099    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-credential-socket" pod "964d70b0-09d8-46d7-98fa-d034aa975096" (UID: "964d70b0-09d8-46d7-98fa-d034aa975096"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:34.152162 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.151975    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-workload-socket" pod "4928c741-8a88-4452-b2ce-f7ed6cff5304" (UID: "4928c741-8a88-4452-b2ce-f7ed6cff5304"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:34.152162 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.152026    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-istio-data" pod "4928c741-8a88-4452-b2ce-f7ed6cff5304" (UID: "4928c741-8a88-4452-b2ce-f7ed6cff5304"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:34.152478 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.152450    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-workload-certs" pod "4928c741-8a88-4452-b2ce-f7ed6cff5304" (UID: "4928c741-8a88-4452-b2ce-f7ed6cff5304"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:34.152564 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.152511    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-credential-socket" pod "4928c741-8a88-4452-b2ce-f7ed6cff5304" (UID: "4928c741-8a88-4452-b2ce-f7ed6cff5304"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:34.153148 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.152892    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/964d70b0-09d8-46d7-98fa-d034aa975096-istiod-ca-cert" pod "964d70b0-09d8-46d7-98fa-d034aa975096" (UID: "964d70b0-09d8-46d7-98fa-d034aa975096"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:49:34.153397 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.153304    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-workload-certs" pod "964d70b0-09d8-46d7-98fa-d034aa975096" (UID: "964d70b0-09d8-46d7-98fa-d034aa975096"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:34.153535 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.153505    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4928c741-8a88-4452-b2ce-f7ed6cff5304-istiod-ca-cert" pod "4928c741-8a88-4452-b2ce-f7ed6cff5304" (UID: "4928c741-8a88-4452-b2ce-f7ed6cff5304"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:49:34.154630 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.154605    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-istio-envoy" pod "964d70b0-09d8-46d7-98fa-d034aa975096" (UID: "964d70b0-09d8-46d7-98fa-d034aa975096"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:34.154758 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.154692    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4928c741-8a88-4452-b2ce-f7ed6cff5304-istio-podinfo" pod "4928c741-8a88-4452-b2ce-f7ed6cff5304" (UID: "4928c741-8a88-4452-b2ce-f7ed6cff5304"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:49:34.154854 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.154834    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/964d70b0-09d8-46d7-98fa-d034aa975096-kube-api-access-27pct" pod "964d70b0-09d8-46d7-98fa-d034aa975096" (UID: "964d70b0-09d8-46d7-98fa-d034aa975096"). InnerVolumeSpecName "kube-api-access-27pct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:34.154981 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.154960    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/964d70b0-09d8-46d7-98fa-d034aa975096-istio-podinfo" pod "964d70b0-09d8-46d7-98fa-d034aa975096" (UID: "964d70b0-09d8-46d7-98fa-d034aa975096"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:49:34.155809 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.155777    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/964d70b0-09d8-46d7-98fa-d034aa975096-istio-token" pod "964d70b0-09d8-46d7-98fa-d034aa975096" (UID: "964d70b0-09d8-46d7-98fa-d034aa975096"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:34.156141 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.156112    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-istio-envoy" pod "4928c741-8a88-4452-b2ce-f7ed6cff5304" (UID: "4928c741-8a88-4452-b2ce-f7ed6cff5304"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:49:34.156208 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.156114    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4928c741-8a88-4452-b2ce-f7ed6cff5304-istio-token" pod "4928c741-8a88-4452-b2ce-f7ed6cff5304" (UID: "4928c741-8a88-4452-b2ce-f7ed6cff5304"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:34.156982 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.156962    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4928c741-8a88-4452-b2ce-f7ed6cff5304-kube-api-access-52wxm" pod "4928c741-8a88-4452-b2ce-f7ed6cff5304" (UID: "4928c741-8a88-4452-b2ce-f7ed6cff5304"). InnerVolumeSpecName "kube-api-access-52wxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:34.252322 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.252289    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27pct\" (UniqueName: \"kubernetes.io/projected/964d70b0-09d8-46d7-98fa-d034aa975096-kube-api-access-27pct\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:34.252322 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.252321    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:34.252527 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.252333    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:34.252527 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.252342    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:34.252527 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.252369    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:34.252527 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.252406    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4928c741-8a88-4452-b2ce-f7ed6cff5304-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:34.252527 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.252415    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:34.252527 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.252424    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52wxm\" (UniqueName: \"kubernetes.io/projected/4928c741-8a88-4452-b2ce-f7ed6cff5304-kube-api-access-52wxm\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:34.252527 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.252432    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/964d70b0-09d8-46d7-98fa-d034aa975096-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:34.252527 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.252440    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4928c741-8a88-4452-b2ce-f7ed6cff5304-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:34.252527 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.252450    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/964d70b0-09d8-46d7-98fa-d034aa975096-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:34.252527 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.252464    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/964d70b0-09d8-46d7-98fa-d034aa975096-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:34.252527 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.252475    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/964d70b0-09d8-46d7-98fa-d034aa975096-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:34.252527 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.252483    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:34.252527 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.252491    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4928c741-8a88-4452-b2ce-f7ed6cff5304-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:34.252527 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.252499    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4928c741-8a88-4452-b2ce-f7ed6cff5304-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:34.538867 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.538826    5087 generic.go:356] "Generic (PLEG): container finished" podID="964d70b0-09d8-46d7-98fa-d034aa975096" containerID="3c32d0925d2cc3dd4b6e41e8c41b544a80ccecd94b07003d32319338d0174695" exitCode=0
Oct 30 21:49:34.539389 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.538894    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"
Oct 30 21:49:34.539389 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.539046    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5" event={"ID":"964d70b0-09d8-46d7-98fa-d034aa975096","Type":"ContainerDied","Data":"3c32d0925d2cc3dd4b6e41e8c41b544a80ccecd94b07003d32319338d0174695"}
Oct 30 21:49:34.539389 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.539097    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5" event={"ID":"964d70b0-09d8-46d7-98fa-d034aa975096","Type":"ContainerDied","Data":"556fd3cceef398ebb259fa9c67ae3a2b3f4bf47ead1aa875f8d9bbc163b5643f"}
Oct 30 21:49:34.539389 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.539128    5087 scope.go:122] "RemoveContainer" containerID="3c32d0925d2cc3dd4b6e41e8c41b544a80ccecd94b07003d32319338d0174695"
Oct 30 21:49:34.543059 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.543030    5087 generic.go:356] "Generic (PLEG): container finished" podID="4928c741-8a88-4452-b2ce-f7ed6cff5304" containerID="4200d982d1b2efc9118d95eb28ec489523a5877766f8063891f88c208ed8be1a" exitCode=0
Oct 30 21:49:34.543408 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.543178    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx" event={"ID":"4928c741-8a88-4452-b2ce-f7ed6cff5304","Type":"ContainerDied","Data":"4200d982d1b2efc9118d95eb28ec489523a5877766f8063891f88c208ed8be1a"}
Oct 30 21:49:34.543408 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.543239    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx" event={"ID":"4928c741-8a88-4452-b2ce-f7ed6cff5304","Type":"ContainerDied","Data":"1b6a8f6972c7358a375d6bf3e09d9bb71b8a6b8cc11b72501078e2ddc1c188fb"}
Oct 30 21:49:34.544536 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.544036    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"
Oct 30 21:49:34.557862 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.557841    5087 scope.go:122] "RemoveContainer" containerID="3c32d0925d2cc3dd4b6e41e8c41b544a80ccecd94b07003d32319338d0174695"
Oct 30 21:49:34.558129 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:49:34.558105    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c32d0925d2cc3dd4b6e41e8c41b544a80ccecd94b07003d32319338d0174695\": container with ID starting with 3c32d0925d2cc3dd4b6e41e8c41b544a80ccecd94b07003d32319338d0174695 not found: ID does not exist" containerID="3c32d0925d2cc3dd4b6e41e8c41b544a80ccecd94b07003d32319338d0174695"
Oct 30 21:49:34.558232 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.558141    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c32d0925d2cc3dd4b6e41e8c41b544a80ccecd94b07003d32319338d0174695"} err="failed to get container status \"3c32d0925d2cc3dd4b6e41e8c41b544a80ccecd94b07003d32319338d0174695\": rpc error: code = NotFound desc = could not find container \"3c32d0925d2cc3dd4b6e41e8c41b544a80ccecd94b07003d32319338d0174695\": container with ID starting with 3c32d0925d2cc3dd4b6e41e8c41b544a80ccecd94b07003d32319338d0174695 not found: ID does not exist"
Oct 30 21:49:34.558232 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.558169    5087 scope.go:122] "RemoveContainer" containerID="4200d982d1b2efc9118d95eb28ec489523a5877766f8063891f88c208ed8be1a"
Oct 30 21:49:34.571077 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.570979    5087 scope.go:122] "RemoveContainer" containerID="4200d982d1b2efc9118d95eb28ec489523a5877766f8063891f88c208ed8be1a"
Oct 30 21:49:34.571523 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:49:34.571487    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4200d982d1b2efc9118d95eb28ec489523a5877766f8063891f88c208ed8be1a\": container with ID starting with 4200d982d1b2efc9118d95eb28ec489523a5877766f8063891f88c208ed8be1a not found: ID does not exist" containerID="4200d982d1b2efc9118d95eb28ec489523a5877766f8063891f88c208ed8be1a"
Oct 30 21:49:34.571657 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.571535    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4200d982d1b2efc9118d95eb28ec489523a5877766f8063891f88c208ed8be1a"} err="failed to get container status \"4200d982d1b2efc9118d95eb28ec489523a5877766f8063891f88c208ed8be1a\": rpc error: code = NotFound desc = could not find container \"4200d982d1b2efc9118d95eb28ec489523a5877766f8063891f88c208ed8be1a\": container with ID starting with 4200d982d1b2efc9118d95eb28ec489523a5877766f8063891f88c208ed8be1a not found: ID does not exist"
Oct 30 21:49:34.706233 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.706197    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"]
Oct 30 21:49:34.714429 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.714405    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5109e7be-q8vpg"
Oct 30 21:49:34.753692 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.753660    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-8xwmx"]
Oct 30 21:49:34.758446 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.758413    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ae089fd-bbb8-49d6-b323-0f8dac4b91d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ae089fd-bbb8-49d6-b323-0f8dac4b91d7-maas-api-openshift-ca\") pod \"3ae089fd-bbb8-49d6-b323-0f8dac4b91d7\" (UID: \"3ae089fd-bbb8-49d6-b323-0f8dac4b91d7\") "
Oct 30 21:49:34.758591 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.758469    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ae089fd-bbb8-49d6-b323-0f8dac4b91d7-kube-api-access-dptx2\" (UniqueName: \"kubernetes.io/projected/3ae089fd-bbb8-49d6-b323-0f8dac4b91d7-kube-api-access-dptx2\") pod \"3ae089fd-bbb8-49d6-b323-0f8dac4b91d7\" (UID: \"3ae089fd-bbb8-49d6-b323-0f8dac4b91d7\") "
Oct 30 21:49:34.758591 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.758495    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ae089fd-bbb8-49d6-b323-0f8dac4b91d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ae089fd-bbb8-49d6-b323-0f8dac4b91d7-maas-api-xks-ca\") pod \"3ae089fd-bbb8-49d6-b323-0f8dac4b91d7\" (UID: \"3ae089fd-bbb8-49d6-b323-0f8dac4b91d7\") "
Oct 30 21:49:34.758873 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.758847    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ae089fd-bbb8-49d6-b323-0f8dac4b91d7-maas-api-openshift-ca" pod "3ae089fd-bbb8-49d6-b323-0f8dac4b91d7" (UID: "3ae089fd-bbb8-49d6-b323-0f8dac4b91d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:49:34.760735 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.760707    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ae089fd-bbb8-49d6-b323-0f8dac4b91d7-kube-api-access-dptx2" pod "3ae089fd-bbb8-49d6-b323-0f8dac4b91d7" (UID: "3ae089fd-bbb8-49d6-b323-0f8dac4b91d7"). InnerVolumeSpecName "kube-api-access-dptx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:49:34.760735 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.760726    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ae089fd-bbb8-49d6-b323-0f8dac4b91d7-maas-api-xks-ca" pod "3ae089fd-bbb8-49d6-b323-0f8dac4b91d7" (UID: "3ae089fd-bbb8-49d6-b323-0f8dac4b91d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:49:34.822702 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.822624    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"]
Oct 30 21:49:34.826895 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.826872    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-rz8s5"]
Oct 30 21:49:34.860112 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.860090    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ae089fd-bbb8-49d6-b323-0f8dac4b91d7-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:34.860112 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.860114    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dptx2\" (UniqueName: \"kubernetes.io/projected/3ae089fd-bbb8-49d6-b323-0f8dac4b91d7-kube-api-access-dptx2\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:34.860266 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:34.860124    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ae089fd-bbb8-49d6-b323-0f8dac4b91d7-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:49:35.550670 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:35.550642    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5109e7be-q8vpg"
Oct 30 21:49:35.550670 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:35.550664    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5109e7be-q8vpg" event={"ID":"3ae089fd-bbb8-49d6-b323-0f8dac4b91d7","Type":"ContainerDied","Data":"1933ab13f86ca5f91ee4c9524df8abb203a6008cecf3944a1e6c841e7dbbf371"}
Oct 30 21:49:35.551288 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:35.550714    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1933ab13f86ca5f91ee4c9524df8abb203a6008cecf3944a1e6c841e7dbbf371"
Oct 30 21:49:35.708624 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:35.708589    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4928c741-8a88-4452-b2ce-f7ed6cff5304" path="/var/lib/kubelet/pods/4928c741-8a88-4452-b2ce-f7ed6cff5304/volumes"
Oct 30 21:49:35.709059 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:49:35.709043    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="964d70b0-09d8-46d7-98fa-d034aa975096" path="/var/lib/kubelet/pods/964d70b0-09d8-46d7-98fa-d034aa975096/volumes"
Oct 30 21:49:47.167527 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:49:47.167487    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:50:03.481075 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:03.481042    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72adc758-rq4xc"]
Oct 30 21:50:03.481573 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:03.481558    5087 cpu_manager.go:414] "containerMap: removing container" podUID="3ae089fd-bbb8-49d6-b323-0f8dac4b91d7" containerName="revoke-keys"
Oct 30 21:50:03.481617 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:03.481574    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ae089fd-bbb8-49d6-b323-0f8dac4b91d7" containerName="revoke-keys"
Oct 30 21:50:03.481617 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:03.481589    5087 cpu_manager.go:414] "containerMap: removing container" podUID="4928c741-8a88-4452-b2ce-f7ed6cff5304" containerName="istio-proxy"
Oct 30 21:50:03.481617 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:03.481595    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4928c741-8a88-4452-b2ce-f7ed6cff5304" containerName="istio-proxy"
Oct 30 21:50:03.481617 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:03.481605    5087 cpu_manager.go:414] "containerMap: removing container" podUID="964d70b0-09d8-46d7-98fa-d034aa975096" containerName="istio-proxy"
Oct 30 21:50:03.481617 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:03.481610    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="964d70b0-09d8-46d7-98fa-d034aa975096" containerName="istio-proxy"
Oct 30 21:50:03.481769 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:03.481678    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ae089fd-bbb8-49d6-b323-0f8dac4b91d7" containerName="revoke-keys"
Oct 30 21:50:03.481769 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:03.481686    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="4928c741-8a88-4452-b2ce-f7ed6cff5304" containerName="istio-proxy"
Oct 30 21:50:03.481769 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:03.481693    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="964d70b0-09d8-46d7-98fa-d034aa975096" containerName="istio-proxy"
Oct 30 21:50:03.484947 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:03.484932    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc758-rq4xc"
Oct 30 21:50:03.562372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:03.562327    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ed3f6ca-e632-4096-ae8b-11420eaaf37a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72adc758-rq4xc\" (UID: \"2ed3f6ca-e632-4096-ae8b-11420eaaf37a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc758-rq4xc"
Oct 30 21:50:03.562528 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:03.562379    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ed3f6ca-e632-4096-ae8b-11420eaaf37a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72adc758-rq4xc\" (UID: \"2ed3f6ca-e632-4096-ae8b-11420eaaf37a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc758-rq4xc"
Oct 30 21:50:03.562528 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:03.562408    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9z2l\" (UniqueName: \"kubernetes.io/projected/2ed3f6ca-e632-4096-ae8b-11420eaaf37a-kube-api-access-r9z2l\") pod \"maas-api-revoke-sub-72adc758-rq4xc\" (UID: \"2ed3f6ca-e632-4096-ae8b-11420eaaf37a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc758-rq4xc"
Oct 30 21:50:03.588200 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:03.588167    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:50:03.588441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:03.588423    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:50:03.590610 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:03.590591    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72adc758-rq4xc"]
Oct 30 21:50:03.663966 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:03.663936    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ed3f6ca-e632-4096-ae8b-11420eaaf37a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72adc758-rq4xc\" (UID: \"2ed3f6ca-e632-4096-ae8b-11420eaaf37a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc758-rq4xc"
Oct 30 21:50:03.664094 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:03.663975    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ed3f6ca-e632-4096-ae8b-11420eaaf37a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72adc758-rq4xc\" (UID: \"2ed3f6ca-e632-4096-ae8b-11420eaaf37a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc758-rq4xc"
Oct 30 21:50:03.664094 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:03.663991    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9z2l\" (UniqueName: \"kubernetes.io/projected/2ed3f6ca-e632-4096-ae8b-11420eaaf37a-kube-api-access-r9z2l\") pod \"maas-api-revoke-sub-72adc758-rq4xc\" (UID: \"2ed3f6ca-e632-4096-ae8b-11420eaaf37a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc758-rq4xc"
Oct 30 21:50:03.664652 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:03.664634    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ed3f6ca-e632-4096-ae8b-11420eaaf37a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72adc758-rq4xc\" (UID: \"2ed3f6ca-e632-4096-ae8b-11420eaaf37a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc758-rq4xc"
Oct 30 21:50:03.666491 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:03.666473    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ed3f6ca-e632-4096-ae8b-11420eaaf37a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72adc758-rq4xc\" (UID: \"2ed3f6ca-e632-4096-ae8b-11420eaaf37a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc758-rq4xc"
Oct 30 21:50:03.721766 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:03.721739    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9z2l\" (UniqueName: \"kubernetes.io/projected/2ed3f6ca-e632-4096-ae8b-11420eaaf37a-kube-api-access-r9z2l\") pod \"maas-api-revoke-sub-72adc758-rq4xc\" (UID: \"2ed3f6ca-e632-4096-ae8b-11420eaaf37a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc758-rq4xc"
Oct 30 21:50:03.798284 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:03.798218    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc758-rq4xc"
Oct 30 21:50:04.178281 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:04.178248    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72adc758-rq4xc"]
Oct 30 21:50:04.179876 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:50:04.179850    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed3f6ca_e632_4096_ae8b_11420eaaf37a.slice/crio-902dab6c2cd1b086aba2513a9d27a981d814b40d43a8838322432b1fd3b21eb8 WatchSource:0}: Error finding container 902dab6c2cd1b086aba2513a9d27a981d814b40d43a8838322432b1fd3b21eb8: Status 404 returned error can't find the container with id 902dab6c2cd1b086aba2513a9d27a981d814b40d43a8838322432b1fd3b21eb8
Oct 30 21:50:04.730528 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:04.730497    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc758-rq4xc" event={"ID":"2ed3f6ca-e632-4096-ae8b-11420eaaf37a","Type":"ContainerStarted","Data":"d72842d63ef93fe019992cc4556956b9f7f8fd2168ccc4339a25d1c87872d3f5"}
Oct 30 21:50:04.730894 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:04.730553    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc758-rq4xc" event={"ID":"2ed3f6ca-e632-4096-ae8b-11420eaaf37a","Type":"ContainerStarted","Data":"902dab6c2cd1b086aba2513a9d27a981d814b40d43a8838322432b1fd3b21eb8"}
Oct 30 21:50:04.847532 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:04.847463    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc758-rq4xc" podStartSLOduration=1.8474452700000001 podStartE2EDuration="1.84744527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:04.846618784 +0000 UTC m=+999.321122767" watchObservedRunningTime="2026-10-30 21:50:04.84744527 +0000 UTC m=+999.321949251"
Oct 30 21:50:05.737794 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:05.737755    5087 generic.go:356] "Generic (PLEG): container finished" podID="2ed3f6ca-e632-4096-ae8b-11420eaaf37a" containerID="d72842d63ef93fe019992cc4556956b9f7f8fd2168ccc4339a25d1c87872d3f5" exitCode=0
Oct 30 21:50:05.738238 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:05.737829    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc758-rq4xc" event={"ID":"2ed3f6ca-e632-4096-ae8b-11420eaaf37a","Type":"ContainerDied","Data":"d72842d63ef93fe019992cc4556956b9f7f8fd2168ccc4339a25d1c87872d3f5"}
Oct 30 21:50:06.908760 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:06.908736    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc758-rq4xc"
Oct 30 21:50:06.997001 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:06.996931    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ed3f6ca-e632-4096-ae8b-11420eaaf37a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ed3f6ca-e632-4096-ae8b-11420eaaf37a-maas-api-xks-ca\") pod \"2ed3f6ca-e632-4096-ae8b-11420eaaf37a\" (UID: \"2ed3f6ca-e632-4096-ae8b-11420eaaf37a\") "
Oct 30 21:50:06.997152 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:06.997027    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ed3f6ca-e632-4096-ae8b-11420eaaf37a-kube-api-access-r9z2l\" (UniqueName: \"kubernetes.io/projected/2ed3f6ca-e632-4096-ae8b-11420eaaf37a-kube-api-access-r9z2l\") pod \"2ed3f6ca-e632-4096-ae8b-11420eaaf37a\" (UID: \"2ed3f6ca-e632-4096-ae8b-11420eaaf37a\") "
Oct 30 21:50:06.997152 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:06.997084    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ed3f6ca-e632-4096-ae8b-11420eaaf37a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ed3f6ca-e632-4096-ae8b-11420eaaf37a-maas-api-openshift-ca\") pod \"2ed3f6ca-e632-4096-ae8b-11420eaaf37a\" (UID: \"2ed3f6ca-e632-4096-ae8b-11420eaaf37a\") "
Oct 30 21:50:06.997485 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:06.997463    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ed3f6ca-e632-4096-ae8b-11420eaaf37a-maas-api-openshift-ca" pod "2ed3f6ca-e632-4096-ae8b-11420eaaf37a" (UID: "2ed3f6ca-e632-4096-ae8b-11420eaaf37a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:06.999256 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:06.999222    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ed3f6ca-e632-4096-ae8b-11420eaaf37a-maas-api-xks-ca" pod "2ed3f6ca-e632-4096-ae8b-11420eaaf37a" (UID: "2ed3f6ca-e632-4096-ae8b-11420eaaf37a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:06.999402 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:06.999321    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ed3f6ca-e632-4096-ae8b-11420eaaf37a-kube-api-access-r9z2l" pod "2ed3f6ca-e632-4096-ae8b-11420eaaf37a" (UID: "2ed3f6ca-e632-4096-ae8b-11420eaaf37a"). InnerVolumeSpecName "kube-api-access-r9z2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:07.098083 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:07.098048    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ed3f6ca-e632-4096-ae8b-11420eaaf37a-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:07.098083 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:07.098075    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9z2l\" (UniqueName: \"kubernetes.io/projected/2ed3f6ca-e632-4096-ae8b-11420eaaf37a-kube-api-access-r9z2l\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:07.098083 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:07.098087    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ed3f6ca-e632-4096-ae8b-11420eaaf37a-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:07.753877 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:07.753846    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc758-rq4xc"
Oct 30 21:50:07.753877 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:07.753867    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72adc758-rq4xc" event={"ID":"2ed3f6ca-e632-4096-ae8b-11420eaaf37a","Type":"ContainerDied","Data":"902dab6c2cd1b086aba2513a9d27a981d814b40d43a8838322432b1fd3b21eb8"}
Oct 30 21:50:07.754110 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:07.753913    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="902dab6c2cd1b086aba2513a9d27a981d814b40d43a8838322432b1fd3b21eb8"
Oct 30 21:50:08.221322 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:08.221282    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e28ef0fa-8vr67"]
Oct 30 21:50:08.221973 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:08.221952    5087 cpu_manager.go:414] "containerMap: removing container" podUID="2ed3f6ca-e632-4096-ae8b-11420eaaf37a" containerName="revoke-keys"
Oct 30 21:50:08.222060 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:08.221975    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ed3f6ca-e632-4096-ae8b-11420eaaf37a" containerName="revoke-keys"
Oct 30 21:50:08.222119 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:08.222100    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ed3f6ca-e632-4096-ae8b-11420eaaf37a" containerName="revoke-keys"
Oct 30 21:50:08.225575 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:08.225554    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ef0fa-8vr67"
Oct 30 21:50:08.279634 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:08.279602    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:50:08.310363 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:08.310318    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff9a88ba-cfeb-4554-8c08-384c1923b19b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e28ef0fa-8vr67\" (UID: \"ff9a88ba-cfeb-4554-8c08-384c1923b19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ef0fa-8vr67"
Oct 30 21:50:08.310545 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:08.310525    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff9a88ba-cfeb-4554-8c08-384c1923b19b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e28ef0fa-8vr67\" (UID: \"ff9a88ba-cfeb-4554-8c08-384c1923b19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ef0fa-8vr67"
Oct 30 21:50:08.310614 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:08.310574    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thdc2\" (UniqueName: \"kubernetes.io/projected/ff9a88ba-cfeb-4554-8c08-384c1923b19b-kube-api-access-thdc2\") pod \"maas-api-revoke-sub-e28ef0fa-8vr67\" (UID: \"ff9a88ba-cfeb-4554-8c08-384c1923b19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ef0fa-8vr67"
Oct 30 21:50:08.328477 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:08.328450    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:50:08.330965 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:08.330937    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e28ef0fa-8vr67"]
Oct 30 21:50:08.411649 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:08.411615    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff9a88ba-cfeb-4554-8c08-384c1923b19b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e28ef0fa-8vr67\" (UID: \"ff9a88ba-cfeb-4554-8c08-384c1923b19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ef0fa-8vr67"
Oct 30 21:50:08.411819 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:08.411665    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thdc2\" (UniqueName: \"kubernetes.io/projected/ff9a88ba-cfeb-4554-8c08-384c1923b19b-kube-api-access-thdc2\") pod \"maas-api-revoke-sub-e28ef0fa-8vr67\" (UID: \"ff9a88ba-cfeb-4554-8c08-384c1923b19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ef0fa-8vr67"
Oct 30 21:50:08.411819 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:08.411739    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff9a88ba-cfeb-4554-8c08-384c1923b19b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e28ef0fa-8vr67\" (UID: \"ff9a88ba-cfeb-4554-8c08-384c1923b19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ef0fa-8vr67"
Oct 30 21:50:08.412442 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:08.412417    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff9a88ba-cfeb-4554-8c08-384c1923b19b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e28ef0fa-8vr67\" (UID: \"ff9a88ba-cfeb-4554-8c08-384c1923b19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ef0fa-8vr67"
Oct 30 21:50:08.414151 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:08.414132    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff9a88ba-cfeb-4554-8c08-384c1923b19b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e28ef0fa-8vr67\" (UID: \"ff9a88ba-cfeb-4554-8c08-384c1923b19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ef0fa-8vr67"
Oct 30 21:50:08.470822 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:08.470792    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thdc2\" (UniqueName: \"kubernetes.io/projected/ff9a88ba-cfeb-4554-8c08-384c1923b19b-kube-api-access-thdc2\") pod \"maas-api-revoke-sub-e28ef0fa-8vr67\" (UID: \"ff9a88ba-cfeb-4554-8c08-384c1923b19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ef0fa-8vr67"
Oct 30 21:50:08.550193 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:08.550168    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ef0fa-8vr67"
Oct 30 21:50:08.934439 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:08.934414    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e28ef0fa-8vr67"]
Oct 30 21:50:08.936299 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:50:08.936275    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff9a88ba_cfeb_4554_8c08_384c1923b19b.slice/crio-84ad8f305fa254a6b47f4111e7d8b3d65c9a0721229973d0e2320444c96109a8 WatchSource:0}: Error finding container 84ad8f305fa254a6b47f4111e7d8b3d65c9a0721229973d0e2320444c96109a8: Status 404 returned error can't find the container with id 84ad8f305fa254a6b47f4111e7d8b3d65c9a0721229973d0e2320444c96109a8
Oct 30 21:50:09.765573 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:09.765537    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ef0fa-8vr67" event={"ID":"ff9a88ba-cfeb-4554-8c08-384c1923b19b","Type":"ContainerStarted","Data":"e3c482c32d62c5cbd505672a982ed6569b7ec55004d08f31319c0efbeb313f62"}
Oct 30 21:50:09.766058 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:09.765592    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ef0fa-8vr67" event={"ID":"ff9a88ba-cfeb-4554-8c08-384c1923b19b","Type":"ContainerStarted","Data":"84ad8f305fa254a6b47f4111e7d8b3d65c9a0721229973d0e2320444c96109a8"}
Oct 30 21:50:09.882200 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:09.882138    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ef0fa-8vr67" podStartSLOduration=1.882121419 podStartE2EDuration="1.882121419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:09.879135692 +0000 UTC m=+1004.353639671" watchObservedRunningTime="2026-10-30 21:50:09.882121419 +0000 UTC m=+1004.356625399"
Oct 30 21:50:10.774184 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:10.774153    5087 generic.go:356] "Generic (PLEG): container finished" podID="ff9a88ba-cfeb-4554-8c08-384c1923b19b" containerID="e3c482c32d62c5cbd505672a982ed6569b7ec55004d08f31319c0efbeb313f62" exitCode=0
Oct 30 21:50:10.774591 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:10.774239    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ef0fa-8vr67" event={"ID":"ff9a88ba-cfeb-4554-8c08-384c1923b19b","Type":"ContainerDied","Data":"e3c482c32d62c5cbd505672a982ed6569b7ec55004d08f31319c0efbeb313f62"}
Oct 30 21:50:11.931901 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:11.931876    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ef0fa-8vr67"
Oct 30 21:50:12.047436 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:12.047364    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff9a88ba-cfeb-4554-8c08-384c1923b19b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff9a88ba-cfeb-4554-8c08-384c1923b19b-maas-api-xks-ca\") pod \"ff9a88ba-cfeb-4554-8c08-384c1923b19b\" (UID: \"ff9a88ba-cfeb-4554-8c08-384c1923b19b\") "
Oct 30 21:50:12.047597 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:12.047458    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff9a88ba-cfeb-4554-8c08-384c1923b19b-kube-api-access-thdc2\" (UniqueName: \"kubernetes.io/projected/ff9a88ba-cfeb-4554-8c08-384c1923b19b-kube-api-access-thdc2\") pod \"ff9a88ba-cfeb-4554-8c08-384c1923b19b\" (UID: \"ff9a88ba-cfeb-4554-8c08-384c1923b19b\") "
Oct 30 21:50:12.047597 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:12.047576    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff9a88ba-cfeb-4554-8c08-384c1923b19b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff9a88ba-cfeb-4554-8c08-384c1923b19b-maas-api-openshift-ca\") pod \"ff9a88ba-cfeb-4554-8c08-384c1923b19b\" (UID: \"ff9a88ba-cfeb-4554-8c08-384c1923b19b\") "
Oct 30 21:50:12.047935 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:12.047909    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff9a88ba-cfeb-4554-8c08-384c1923b19b-maas-api-openshift-ca" pod "ff9a88ba-cfeb-4554-8c08-384c1923b19b" (UID: "ff9a88ba-cfeb-4554-8c08-384c1923b19b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:12.049844 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:12.049813    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff9a88ba-cfeb-4554-8c08-384c1923b19b-maas-api-xks-ca" pod "ff9a88ba-cfeb-4554-8c08-384c1923b19b" (UID: "ff9a88ba-cfeb-4554-8c08-384c1923b19b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:12.050095 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:12.050069    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff9a88ba-cfeb-4554-8c08-384c1923b19b-kube-api-access-thdc2" pod "ff9a88ba-cfeb-4554-8c08-384c1923b19b" (UID: "ff9a88ba-cfeb-4554-8c08-384c1923b19b"). InnerVolumeSpecName "kube-api-access-thdc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:12.148324 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:12.148295    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff9a88ba-cfeb-4554-8c08-384c1923b19b-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:12.148324 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:12.148318    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff9a88ba-cfeb-4554-8c08-384c1923b19b-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:12.148324 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:12.148329    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thdc2\" (UniqueName: \"kubernetes.io/projected/ff9a88ba-cfeb-4554-8c08-384c1923b19b-kube-api-access-thdc2\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:12.788042 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:12.788011    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ef0fa-8vr67"
Oct 30 21:50:12.788042 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:12.788024    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e28ef0fa-8vr67" event={"ID":"ff9a88ba-cfeb-4554-8c08-384c1923b19b","Type":"ContainerDied","Data":"84ad8f305fa254a6b47f4111e7d8b3d65c9a0721229973d0e2320444c96109a8"}
Oct 30 21:50:12.788252 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:12.788078    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84ad8f305fa254a6b47f4111e7d8b3d65c9a0721229973d0e2320444c96109a8"
Oct 30 21:50:14.214568 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:14.214530    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da29607a-hqpxr"]
Oct 30 21:50:14.215019 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:14.215005    5087 cpu_manager.go:414] "containerMap: removing container" podUID="ff9a88ba-cfeb-4554-8c08-384c1923b19b" containerName="revoke-keys"
Oct 30 21:50:14.215066 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:14.215020    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff9a88ba-cfeb-4554-8c08-384c1923b19b" containerName="revoke-keys"
Oct 30 21:50:14.215102 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:14.215090    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff9a88ba-cfeb-4554-8c08-384c1923b19b" containerName="revoke-keys"
Oct 30 21:50:14.218423 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:14.218401    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da29607a-hqpxr"
Oct 30 21:50:14.272506 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:14.272476    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc76d42b-2efe-4f83-8ebe-6b7d54bf455c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da29607a-hqpxr\" (UID: \"cc76d42b-2efe-4f83-8ebe-6b7d54bf455c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da29607a-hqpxr"
Oct 30 21:50:14.272831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:14.272800    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc76d42b-2efe-4f83-8ebe-6b7d54bf455c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da29607a-hqpxr\" (UID: \"cc76d42b-2efe-4f83-8ebe-6b7d54bf455c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da29607a-hqpxr"
Oct 30 21:50:14.272980 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:14.272842    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv749\" (UniqueName: \"kubernetes.io/projected/cc76d42b-2efe-4f83-8ebe-6b7d54bf455c-kube-api-access-dv749\") pod \"maas-api-revoke-sub-da29607a-hqpxr\" (UID: \"cc76d42b-2efe-4f83-8ebe-6b7d54bf455c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da29607a-hqpxr"
Oct 30 21:50:14.321609 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:14.321579    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:50:14.321863 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:14.321843    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:50:14.324258 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:14.324236    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da29607a-hqpxr"]
Oct 30 21:50:14.373692 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:14.373665    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc76d42b-2efe-4f83-8ebe-6b7d54bf455c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da29607a-hqpxr\" (UID: \"cc76d42b-2efe-4f83-8ebe-6b7d54bf455c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da29607a-hqpxr"
Oct 30 21:50:14.373885 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:14.373728    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc76d42b-2efe-4f83-8ebe-6b7d54bf455c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da29607a-hqpxr\" (UID: \"cc76d42b-2efe-4f83-8ebe-6b7d54bf455c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da29607a-hqpxr"
Oct 30 21:50:14.373885 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:14.373753    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv749\" (UniqueName: \"kubernetes.io/projected/cc76d42b-2efe-4f83-8ebe-6b7d54bf455c-kube-api-access-dv749\") pod \"maas-api-revoke-sub-da29607a-hqpxr\" (UID: \"cc76d42b-2efe-4f83-8ebe-6b7d54bf455c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da29607a-hqpxr"
Oct 30 21:50:14.374624 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:14.374600    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc76d42b-2efe-4f83-8ebe-6b7d54bf455c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da29607a-hqpxr\" (UID: \"cc76d42b-2efe-4f83-8ebe-6b7d54bf455c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da29607a-hqpxr"
Oct 30 21:50:14.376416 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:14.376391    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc76d42b-2efe-4f83-8ebe-6b7d54bf455c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da29607a-hqpxr\" (UID: \"cc76d42b-2efe-4f83-8ebe-6b7d54bf455c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da29607a-hqpxr"
Oct 30 21:50:14.430762 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:14.430732    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv749\" (UniqueName: \"kubernetes.io/projected/cc76d42b-2efe-4f83-8ebe-6b7d54bf455c-kube-api-access-dv749\") pod \"maas-api-revoke-sub-da29607a-hqpxr\" (UID: \"cc76d42b-2efe-4f83-8ebe-6b7d54bf455c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da29607a-hqpxr"
Oct 30 21:50:14.532808 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:14.532781    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da29607a-hqpxr"
Oct 30 21:50:14.911872 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:14.911847    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da29607a-hqpxr"]
Oct 30 21:50:14.913710 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:50:14.913683    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc76d42b_2efe_4f83_8ebe_6b7d54bf455c.slice/crio-6cf5cda9a046bc250abcd00774bbc0b97705ab520561cf3ec4d7e4f507a3d6ef WatchSource:0}: Error finding container 6cf5cda9a046bc250abcd00774bbc0b97705ab520561cf3ec4d7e4f507a3d6ef: Status 404 returned error can't find the container with id 6cf5cda9a046bc250abcd00774bbc0b97705ab520561cf3ec4d7e4f507a3d6ef
Oct 30 21:50:15.806422 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:15.806379    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da29607a-hqpxr" event={"ID":"cc76d42b-2efe-4f83-8ebe-6b7d54bf455c","Type":"ContainerStarted","Data":"7ef8a984107848606d82df671d3d0ae0521285d921b2af65b90960ea6cd9bd14"}
Oct 30 21:50:15.806870 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:15.806453    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da29607a-hqpxr" event={"ID":"cc76d42b-2efe-4f83-8ebe-6b7d54bf455c","Type":"ContainerStarted","Data":"6cf5cda9a046bc250abcd00774bbc0b97705ab520561cf3ec4d7e4f507a3d6ef"}
Oct 30 21:50:15.930481 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:15.930430    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da29607a-hqpxr" podStartSLOduration=1.930416463 podStartE2EDuration="1.930416463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:15.930204669 +0000 UTC m=+1010.404708649" watchObservedRunningTime="2026-10-30 21:50:15.930416463 +0000 UTC m=+1010.404920456"
Oct 30 21:50:16.813241 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:16.813212    5087 generic.go:356] "Generic (PLEG): container finished" podID="cc76d42b-2efe-4f83-8ebe-6b7d54bf455c" containerID="7ef8a984107848606d82df671d3d0ae0521285d921b2af65b90960ea6cd9bd14" exitCode=0
Oct 30 21:50:16.813797 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:16.813300    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da29607a-hqpxr" event={"ID":"cc76d42b-2efe-4f83-8ebe-6b7d54bf455c","Type":"ContainerDied","Data":"7ef8a984107848606d82df671d3d0ae0521285d921b2af65b90960ea6cd9bd14"}
Oct 30 21:50:17.166772 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:50:17.166687    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:50:17.958640 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:17.958618    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da29607a-hqpxr"
Oct 30 21:50:18.008679 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:18.008651    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc76d42b-2efe-4f83-8ebe-6b7d54bf455c-kube-api-access-dv749\" (UniqueName: \"kubernetes.io/projected/cc76d42b-2efe-4f83-8ebe-6b7d54bf455c-kube-api-access-dv749\") pod \"cc76d42b-2efe-4f83-8ebe-6b7d54bf455c\" (UID: \"cc76d42b-2efe-4f83-8ebe-6b7d54bf455c\") "
Oct 30 21:50:18.008790 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:18.008701    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc76d42b-2efe-4f83-8ebe-6b7d54bf455c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc76d42b-2efe-4f83-8ebe-6b7d54bf455c-maas-api-xks-ca\") pod \"cc76d42b-2efe-4f83-8ebe-6b7d54bf455c\" (UID: \"cc76d42b-2efe-4f83-8ebe-6b7d54bf455c\") "
Oct 30 21:50:18.008790 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:18.008781    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc76d42b-2efe-4f83-8ebe-6b7d54bf455c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc76d42b-2efe-4f83-8ebe-6b7d54bf455c-maas-api-openshift-ca\") pod \"cc76d42b-2efe-4f83-8ebe-6b7d54bf455c\" (UID: \"cc76d42b-2efe-4f83-8ebe-6b7d54bf455c\") "
Oct 30 21:50:18.009110 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:18.009086    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc76d42b-2efe-4f83-8ebe-6b7d54bf455c-maas-api-openshift-ca" pod "cc76d42b-2efe-4f83-8ebe-6b7d54bf455c" (UID: "cc76d42b-2efe-4f83-8ebe-6b7d54bf455c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:18.010968 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:18.010941    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc76d42b-2efe-4f83-8ebe-6b7d54bf455c-maas-api-xks-ca" pod "cc76d42b-2efe-4f83-8ebe-6b7d54bf455c" (UID: "cc76d42b-2efe-4f83-8ebe-6b7d54bf455c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:18.011044 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:18.010998    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc76d42b-2efe-4f83-8ebe-6b7d54bf455c-kube-api-access-dv749" pod "cc76d42b-2efe-4f83-8ebe-6b7d54bf455c" (UID: "cc76d42b-2efe-4f83-8ebe-6b7d54bf455c"). InnerVolumeSpecName "kube-api-access-dv749". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:18.110471 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:18.110405    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv749\" (UniqueName: \"kubernetes.io/projected/cc76d42b-2efe-4f83-8ebe-6b7d54bf455c-kube-api-access-dv749\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:18.110471 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:18.110432    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc76d42b-2efe-4f83-8ebe-6b7d54bf455c-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:18.110471 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:18.110442    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc76d42b-2efe-4f83-8ebe-6b7d54bf455c-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:18.824825 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:18.824798    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da29607a-hqpxr"
Oct 30 21:50:18.824988 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:18.824837    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da29607a-hqpxr" event={"ID":"cc76d42b-2efe-4f83-8ebe-6b7d54bf455c","Type":"ContainerDied","Data":"6cf5cda9a046bc250abcd00774bbc0b97705ab520561cf3ec4d7e4f507a3d6ef"}
Oct 30 21:50:18.824988 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:18.824886    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cf5cda9a046bc250abcd00774bbc0b97705ab520561cf3ec4d7e4f507a3d6ef"
Oct 30 21:50:20.449575 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:20.449543    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82332eeb-nnglk"]
Oct 30 21:50:20.450072 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:20.450056    5087 cpu_manager.go:414] "containerMap: removing container" podUID="cc76d42b-2efe-4f83-8ebe-6b7d54bf455c" containerName="revoke-keys"
Oct 30 21:50:20.450118 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:20.450074    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc76d42b-2efe-4f83-8ebe-6b7d54bf455c" containerName="revoke-keys"
Oct 30 21:50:20.450189 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:20.450180    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc76d42b-2efe-4f83-8ebe-6b7d54bf455c" containerName="revoke-keys"
Oct 30 21:50:20.453578 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:20.453561    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82332eeb-nnglk"
Oct 30 21:50:20.506321 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:20.506297    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:50:20.532988 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:20.532959    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/960bcf64-057c-4ef8-9d89-d61c54237828-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82332eeb-nnglk\" (UID: \"960bcf64-057c-4ef8-9d89-d61c54237828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82332eeb-nnglk"
Oct 30 21:50:20.533132 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:20.533031    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vj6d\" (UniqueName: \"kubernetes.io/projected/960bcf64-057c-4ef8-9d89-d61c54237828-kube-api-access-5vj6d\") pod \"maas-api-revoke-sub-82332eeb-nnglk\" (UID: \"960bcf64-057c-4ef8-9d89-d61c54237828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82332eeb-nnglk"
Oct 30 21:50:20.533309 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:20.533281    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/960bcf64-057c-4ef8-9d89-d61c54237828-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82332eeb-nnglk\" (UID: \"960bcf64-057c-4ef8-9d89-d61c54237828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82332eeb-nnglk"
Oct 30 21:50:20.533416 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:20.533391    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82332eeb-nnglk"]
Oct 30 21:50:20.555593 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:20.555573    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:50:20.634205 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:20.634174    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vj6d\" (UniqueName: \"kubernetes.io/projected/960bcf64-057c-4ef8-9d89-d61c54237828-kube-api-access-5vj6d\") pod \"maas-api-revoke-sub-82332eeb-nnglk\" (UID: \"960bcf64-057c-4ef8-9d89-d61c54237828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82332eeb-nnglk"
Oct 30 21:50:20.634342 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:20.634308    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/960bcf64-057c-4ef8-9d89-d61c54237828-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82332eeb-nnglk\" (UID: \"960bcf64-057c-4ef8-9d89-d61c54237828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82332eeb-nnglk"
Oct 30 21:50:20.634342 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:20.634334    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/960bcf64-057c-4ef8-9d89-d61c54237828-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82332eeb-nnglk\" (UID: \"960bcf64-057c-4ef8-9d89-d61c54237828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82332eeb-nnglk"
Oct 30 21:50:20.634978 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:20.634956    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/960bcf64-057c-4ef8-9d89-d61c54237828-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82332eeb-nnglk\" (UID: \"960bcf64-057c-4ef8-9d89-d61c54237828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82332eeb-nnglk"
Oct 30 21:50:20.636853 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:20.636833    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/960bcf64-057c-4ef8-9d89-d61c54237828-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82332eeb-nnglk\" (UID: \"960bcf64-057c-4ef8-9d89-d61c54237828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82332eeb-nnglk"
Oct 30 21:50:20.691174 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:20.691144    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vj6d\" (UniqueName: \"kubernetes.io/projected/960bcf64-057c-4ef8-9d89-d61c54237828-kube-api-access-5vj6d\") pod \"maas-api-revoke-sub-82332eeb-nnglk\" (UID: \"960bcf64-057c-4ef8-9d89-d61c54237828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82332eeb-nnglk"
Oct 30 21:50:20.766072 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:20.766041    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82332eeb-nnglk"
Oct 30 21:50:21.140594 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:21.140565    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82332eeb-nnglk"]
Oct 30 21:50:21.142531 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:50:21.142503    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod960bcf64_057c_4ef8_9d89_d61c54237828.slice/crio-04a2c2177233c532ffd8e4146b72366306f4f4dc27a62a2058a05db71d175c7e WatchSource:0}: Error finding container 04a2c2177233c532ffd8e4146b72366306f4f4dc27a62a2058a05db71d175c7e: Status 404 returned error can't find the container with id 04a2c2177233c532ffd8e4146b72366306f4f4dc27a62a2058a05db71d175c7e
Oct 30 21:50:21.848362 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:21.848312    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82332eeb-nnglk" event={"ID":"960bcf64-057c-4ef8-9d89-d61c54237828","Type":"ContainerStarted","Data":"7de756bb94d594a8d03becc21028f78e5dbc3f5e9d51006ac095a4d09e8a4d2c"}
Oct 30 21:50:21.848752 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:21.848390    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82332eeb-nnglk" event={"ID":"960bcf64-057c-4ef8-9d89-d61c54237828","Type":"ContainerStarted","Data":"04a2c2177233c532ffd8e4146b72366306f4f4dc27a62a2058a05db71d175c7e"}
Oct 30 21:50:21.963024 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:21.962972    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82332eeb-nnglk" podStartSLOduration=1.96295993 podStartE2EDuration="1.96295993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:21.961712339 +0000 UTC m=+1016.436216317" watchObservedRunningTime="2026-10-30 21:50:21.96295993 +0000 UTC m=+1016.437463909"
Oct 30 21:50:22.857678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:22.857647    5087 generic.go:356] "Generic (PLEG): container finished" podID="960bcf64-057c-4ef8-9d89-d61c54237828" containerID="7de756bb94d594a8d03becc21028f78e5dbc3f5e9d51006ac095a4d09e8a4d2c" exitCode=0
Oct 30 21:50:22.858182 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:22.857712    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82332eeb-nnglk" event={"ID":"960bcf64-057c-4ef8-9d89-d61c54237828","Type":"ContainerDied","Data":"7de756bb94d594a8d03becc21028f78e5dbc3f5e9d51006ac095a4d09e8a4d2c"}
Oct 30 21:50:24.021681 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:24.021658    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82332eeb-nnglk"
Oct 30 21:50:24.177887 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:24.177810    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/960bcf64-057c-4ef8-9d89-d61c54237828-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/960bcf64-057c-4ef8-9d89-d61c54237828-maas-api-openshift-ca\") pod \"960bcf64-057c-4ef8-9d89-d61c54237828\" (UID: \"960bcf64-057c-4ef8-9d89-d61c54237828\") "
Oct 30 21:50:24.178058 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:24.177906    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/960bcf64-057c-4ef8-9d89-d61c54237828-kube-api-access-5vj6d\" (UniqueName: \"kubernetes.io/projected/960bcf64-057c-4ef8-9d89-d61c54237828-kube-api-access-5vj6d\") pod \"960bcf64-057c-4ef8-9d89-d61c54237828\" (UID: \"960bcf64-057c-4ef8-9d89-d61c54237828\") "
Oct 30 21:50:24.178058 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:24.177963    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/960bcf64-057c-4ef8-9d89-d61c54237828-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/960bcf64-057c-4ef8-9d89-d61c54237828-maas-api-xks-ca\") pod \"960bcf64-057c-4ef8-9d89-d61c54237828\" (UID: \"960bcf64-057c-4ef8-9d89-d61c54237828\") "
Oct 30 21:50:24.178166 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:24.178146    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/960bcf64-057c-4ef8-9d89-d61c54237828-maas-api-openshift-ca" pod "960bcf64-057c-4ef8-9d89-d61c54237828" (UID: "960bcf64-057c-4ef8-9d89-d61c54237828"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:24.178296 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:24.178279    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/960bcf64-057c-4ef8-9d89-d61c54237828-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:24.180042 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:24.180022    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/960bcf64-057c-4ef8-9d89-d61c54237828-maas-api-xks-ca" pod "960bcf64-057c-4ef8-9d89-d61c54237828" (UID: "960bcf64-057c-4ef8-9d89-d61c54237828"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:24.180174 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:24.180154    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/960bcf64-057c-4ef8-9d89-d61c54237828-kube-api-access-5vj6d" pod "960bcf64-057c-4ef8-9d89-d61c54237828" (UID: "960bcf64-057c-4ef8-9d89-d61c54237828"). InnerVolumeSpecName "kube-api-access-5vj6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:24.279281 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:24.279246    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vj6d\" (UniqueName: \"kubernetes.io/projected/960bcf64-057c-4ef8-9d89-d61c54237828-kube-api-access-5vj6d\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:24.279281 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:24.279276    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/960bcf64-057c-4ef8-9d89-d61c54237828-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:24.869019 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:24.868985    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82332eeb-nnglk" event={"ID":"960bcf64-057c-4ef8-9d89-d61c54237828","Type":"ContainerDied","Data":"04a2c2177233c532ffd8e4146b72366306f4f4dc27a62a2058a05db71d175c7e"}
Oct 30 21:50:24.869191 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:24.868996    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82332eeb-nnglk"
Oct 30 21:50:24.869191 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:24.869045    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04a2c2177233c532ffd8e4146b72366306f4f4dc27a62a2058a05db71d175c7e"
Oct 30 21:50:25.776793 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:25.776760    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe21caf8-29dhc"]
Oct 30 21:50:25.777436 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:25.777416    5087 cpu_manager.go:414] "containerMap: removing container" podUID="960bcf64-057c-4ef8-9d89-d61c54237828" containerName="revoke-keys"
Oct 30 21:50:25.777545 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:25.777437    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="960bcf64-057c-4ef8-9d89-d61c54237828" containerName="revoke-keys"
Oct 30 21:50:25.777617 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:25.777604    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="960bcf64-057c-4ef8-9d89-d61c54237828" containerName="revoke-keys"
Oct 30 21:50:25.781095 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:25.781074    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe21caf8-29dhc"
Oct 30 21:50:25.833546 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:25.833517    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:50:25.837228 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:25.837198    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe21caf8-29dhc"]
Oct 30 21:50:25.883056 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:25.883023    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:50:25.898277 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:25.898249    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f6ba80f-83df-4f15-84e0-a63607744fcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe21caf8-29dhc\" (UID: \"4f6ba80f-83df-4f15-84e0-a63607744fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe21caf8-29dhc"
Oct 30 21:50:25.898412 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:25.898321    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f6ba80f-83df-4f15-84e0-a63607744fcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe21caf8-29dhc\" (UID: \"4f6ba80f-83df-4f15-84e0-a63607744fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe21caf8-29dhc"
Oct 30 21:50:25.898496 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:25.898476    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvnvr\" (UniqueName: \"kubernetes.io/projected/4f6ba80f-83df-4f15-84e0-a63607744fcb-kube-api-access-qvnvr\") pod \"maas-api-revoke-sub-fe21caf8-29dhc\" (UID: \"4f6ba80f-83df-4f15-84e0-a63607744fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe21caf8-29dhc"
Oct 30 21:50:25.999908 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:25.999869    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f6ba80f-83df-4f15-84e0-a63607744fcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe21caf8-29dhc\" (UID: \"4f6ba80f-83df-4f15-84e0-a63607744fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe21caf8-29dhc"
Oct 30 21:50:26.000099 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:25.999954    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvnvr\" (UniqueName: \"kubernetes.io/projected/4f6ba80f-83df-4f15-84e0-a63607744fcb-kube-api-access-qvnvr\") pod \"maas-api-revoke-sub-fe21caf8-29dhc\" (UID: \"4f6ba80f-83df-4f15-84e0-a63607744fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe21caf8-29dhc"
Oct 30 21:50:26.000099 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:26.000025    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f6ba80f-83df-4f15-84e0-a63607744fcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe21caf8-29dhc\" (UID: \"4f6ba80f-83df-4f15-84e0-a63607744fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe21caf8-29dhc"
Oct 30 21:50:26.000562 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:26.000542    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f6ba80f-83df-4f15-84e0-a63607744fcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe21caf8-29dhc\" (UID: \"4f6ba80f-83df-4f15-84e0-a63607744fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe21caf8-29dhc"
Oct 30 21:50:26.002587 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:26.002559    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f6ba80f-83df-4f15-84e0-a63607744fcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe21caf8-29dhc\" (UID: \"4f6ba80f-83df-4f15-84e0-a63607744fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe21caf8-29dhc"
Oct 30 21:50:26.057141 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:26.057070    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvnvr\" (UniqueName: \"kubernetes.io/projected/4f6ba80f-83df-4f15-84e0-a63607744fcb-kube-api-access-qvnvr\") pod \"maas-api-revoke-sub-fe21caf8-29dhc\" (UID: \"4f6ba80f-83df-4f15-84e0-a63607744fcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe21caf8-29dhc"
Oct 30 21:50:26.095075 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:26.095051    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe21caf8-29dhc"
Oct 30 21:50:26.480233 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:26.480205    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe21caf8-29dhc"]
Oct 30 21:50:26.481426 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:50:26.481396    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f6ba80f_83df_4f15_84e0_a63607744fcb.slice/crio-47886b023a919a53f096c89cd6b2fdd1d4cebf4340436c589c6e0e04846408e8 WatchSource:0}: Error finding container 47886b023a919a53f096c89cd6b2fdd1d4cebf4340436c589c6e0e04846408e8: Status 404 returned error can't find the container with id 47886b023a919a53f096c89cd6b2fdd1d4cebf4340436c589c6e0e04846408e8
Oct 30 21:50:26.881986 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:26.881952    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe21caf8-29dhc" event={"ID":"4f6ba80f-83df-4f15-84e0-a63607744fcb","Type":"ContainerStarted","Data":"a5058454eca51f93d711cf1b7645ef3cab56c0e6b6a122c939612e43f12fb5cb"}
Oct 30 21:50:26.882481 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:26.882006    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe21caf8-29dhc" event={"ID":"4f6ba80f-83df-4f15-84e0-a63607744fcb","Type":"ContainerStarted","Data":"47886b023a919a53f096c89cd6b2fdd1d4cebf4340436c589c6e0e04846408e8"}
Oct 30 21:50:26.997104 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:26.996999    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe21caf8-29dhc" podStartSLOduration=1.996986539 podStartE2EDuration="1.996986539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:26.994500739 +0000 UTC m=+1021.469004718" watchObservedRunningTime="2026-10-30 21:50:26.996986539 +0000 UTC m=+1021.471490518"
Oct 30 21:50:27.889280 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:27.889248    5087 generic.go:356] "Generic (PLEG): container finished" podID="4f6ba80f-83df-4f15-84e0-a63607744fcb" containerID="a5058454eca51f93d711cf1b7645ef3cab56c0e6b6a122c939612e43f12fb5cb" exitCode=0
Oct 30 21:50:27.889280 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:27.889283    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe21caf8-29dhc" event={"ID":"4f6ba80f-83df-4f15-84e0-a63607744fcb","Type":"ContainerDied","Data":"a5058454eca51f93d711cf1b7645ef3cab56c0e6b6a122c939612e43f12fb5cb"}
Oct 30 21:50:28.141165 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.141085    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"]
Oct 30 21:50:28.145028 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.145009    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:28.197102 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.197082    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-bc009f-openshift-default-dockercfg-4jwnk\""
Oct 30 21:50:28.204763 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.204738    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"]
Oct 30 21:50:28.322369 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.322305    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-istio-data\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:28.322543 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.322442    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-workload-socket\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:28.322543 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.322517    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-workload-certs\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:28.322631 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.322554    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/284b12e8-9e86-4406-8494-360e02159b74-istiod-ca-cert\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:28.322631 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.322625    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-istio-envoy\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:28.322702 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.322648    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/284b12e8-9e86-4406-8494-360e02159b74-istio-token\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:28.322702 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.322677    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/284b12e8-9e86-4406-8494-360e02159b74-istio-podinfo\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:28.322780 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.322754    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7m46\" (UniqueName: \"kubernetes.io/projected/284b12e8-9e86-4406-8494-360e02159b74-kube-api-access-k7m46\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:28.322813 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.322793    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-credential-socket\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:28.424103 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.424013    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/284b12e8-9e86-4406-8494-360e02159b74-istio-podinfo\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:28.424103 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.424082    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7m46\" (UniqueName: \"kubernetes.io/projected/284b12e8-9e86-4406-8494-360e02159b74-kube-api-access-k7m46\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:28.424324 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.424113    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-credential-socket\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:28.424324 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.424156    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-istio-data\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:28.424324 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.424213    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-workload-socket\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:28.424529 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.424360    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-workload-certs\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:28.424529 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.424430    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/284b12e8-9e86-4406-8494-360e02159b74-istiod-ca-cert\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:28.424529 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.424489    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-istio-envoy\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:28.424529 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.424519    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/284b12e8-9e86-4406-8494-360e02159b74-istio-token\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:28.424724 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.424606    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-workload-socket\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:28.424724 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.424701    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-istio-data\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:28.424828 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.424767    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-credential-socket\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:28.424879 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.424852    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-workload-certs\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:28.425120 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.425100    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/284b12e8-9e86-4406-8494-360e02159b74-istiod-ca-cert\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:28.426666 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.426644    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/284b12e8-9e86-4406-8494-360e02159b74-istio-podinfo\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:28.427054 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.427034    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-istio-envoy\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:28.481127 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.481094    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7m46\" (UniqueName: \"kubernetes.io/projected/284b12e8-9e86-4406-8494-360e02159b74-kube-api-access-k7m46\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:28.481267 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.481215    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/284b12e8-9e86-4406-8494-360e02159b74-istio-token\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:28.760682 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:28.760649    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:29.036592 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:29.036558    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe21caf8-29dhc"
Oct 30 21:50:29.132964 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:29.132932    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f6ba80f-83df-4f15-84e0-a63607744fcb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f6ba80f-83df-4f15-84e0-a63607744fcb-maas-api-xks-ca\") pod \"4f6ba80f-83df-4f15-84e0-a63607744fcb\" (UID: \"4f6ba80f-83df-4f15-84e0-a63607744fcb\") "
Oct 30 21:50:29.133146 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:29.133098    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f6ba80f-83df-4f15-84e0-a63607744fcb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f6ba80f-83df-4f15-84e0-a63607744fcb-maas-api-openshift-ca\") pod \"4f6ba80f-83df-4f15-84e0-a63607744fcb\" (UID: \"4f6ba80f-83df-4f15-84e0-a63607744fcb\") "
Oct 30 21:50:29.133146 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:29.133138    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f6ba80f-83df-4f15-84e0-a63607744fcb-kube-api-access-qvnvr\" (UniqueName: \"kubernetes.io/projected/4f6ba80f-83df-4f15-84e0-a63607744fcb-kube-api-access-qvnvr\") pod \"4f6ba80f-83df-4f15-84e0-a63607744fcb\" (UID: \"4f6ba80f-83df-4f15-84e0-a63607744fcb\") "
Oct 30 21:50:29.133552 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:29.133519    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f6ba80f-83df-4f15-84e0-a63607744fcb-maas-api-openshift-ca" pod "4f6ba80f-83df-4f15-84e0-a63607744fcb" (UID: "4f6ba80f-83df-4f15-84e0-a63607744fcb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:29.135106 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:29.135078    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f6ba80f-83df-4f15-84e0-a63607744fcb-maas-api-xks-ca" pod "4f6ba80f-83df-4f15-84e0-a63607744fcb" (UID: "4f6ba80f-83df-4f15-84e0-a63607744fcb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:29.135230 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:29.135214    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f6ba80f-83df-4f15-84e0-a63607744fcb-kube-api-access-qvnvr" pod "4f6ba80f-83df-4f15-84e0-a63607744fcb" (UID: "4f6ba80f-83df-4f15-84e0-a63607744fcb"). InnerVolumeSpecName "kube-api-access-qvnvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:29.197945 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:50:29.197918    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod284b12e8_9e86_4406_8494_360e02159b74.slice/crio-dc94bb436e0fbbd68b7849bfda1edc66bf1348e8490f95c0a83d83e636c333e9 WatchSource:0}: Error finding container dc94bb436e0fbbd68b7849bfda1edc66bf1348e8490f95c0a83d83e636c333e9: Status 404 returned error can't find the container with id dc94bb436e0fbbd68b7849bfda1edc66bf1348e8490f95c0a83d83e636c333e9
Oct 30 21:50:29.200780 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:29.200744    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:50:29.200891 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:29.200830    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:50:29.200891 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:29.200871    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:50:29.212946 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:29.212923    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"]
Oct 30 21:50:29.233864 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:29.233841    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f6ba80f-83df-4f15-84e0-a63607744fcb-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:29.233864 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:29.233865    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvnvr\" (UniqueName: \"kubernetes.io/projected/4f6ba80f-83df-4f15-84e0-a63607744fcb-kube-api-access-qvnvr\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:29.234022 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:29.233874    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f6ba80f-83df-4f15-84e0-a63607744fcb-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:29.903962 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:29.903932    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe21caf8-29dhc"
Oct 30 21:50:29.904138 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:29.903962    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe21caf8-29dhc" event={"ID":"4f6ba80f-83df-4f15-84e0-a63607744fcb","Type":"ContainerDied","Data":"47886b023a919a53f096c89cd6b2fdd1d4cebf4340436c589c6e0e04846408e8"}
Oct 30 21:50:29.904138 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:29.904012    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47886b023a919a53f096c89cd6b2fdd1d4cebf4340436c589c6e0e04846408e8"
Oct 30 21:50:29.906400 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:29.905883    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj" event={"ID":"284b12e8-9e86-4406-8494-360e02159b74","Type":"ContainerStarted","Data":"07e6718e2230c96163eb10433b31764ff49c04c71a9bd4c4e51ef35518d5a11b"}
Oct 30 21:50:29.906400 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:29.905935    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj" event={"ID":"284b12e8-9e86-4406-8494-360e02159b74","Type":"ContainerStarted","Data":"dc94bb436e0fbbd68b7849bfda1edc66bf1348e8490f95c0a83d83e636c333e9"}
Oct 30 21:50:30.026491 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:30.026430    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj" podStartSLOduration=2.026412369 podStartE2EDuration="2.026412369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:30.024484074 +0000 UTC m=+1024.498988047" watchObservedRunningTime="2026-10-30 21:50:30.026412369 +0000 UTC m=+1024.500916349"
Oct 30 21:50:30.764735 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:30.764704    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:30.769096 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:30.769071    5087 patch_prober.go:29] interesting pod/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 21:50:30.769209 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:30.769122    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj" podUID="284b12e8-9e86-4406-8494-360e02159b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 21:50:31.762777 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:31.762739    5087 patch_prober.go:29] interesting pod/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 21:50:31.762988 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:31.762806    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj" podUID="284b12e8-9e86-4406-8494-360e02159b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 21:50:32.768376 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:32.764746    5087 patch_prober.go:29] interesting pod/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 21:50:32.768376 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:32.764812    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj" podUID="284b12e8-9e86-4406-8494-360e02159b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 21:50:33.528694 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:33.528663    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d39ad088-f9dfn"]
Oct 30 21:50:33.529173 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:33.529157    5087 cpu_manager.go:414] "containerMap: removing container" podUID="4f6ba80f-83df-4f15-84e0-a63607744fcb" containerName="revoke-keys"
Oct 30 21:50:33.529228 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:33.529174    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f6ba80f-83df-4f15-84e0-a63607744fcb" containerName="revoke-keys"
Oct 30 21:50:33.529284 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:33.529274    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f6ba80f-83df-4f15-84e0-a63607744fcb" containerName="revoke-keys"
Oct 30 21:50:33.532612 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:33.532593    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d39ad088-f9dfn"
Oct 30 21:50:33.587003 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:33.586955    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:50:33.598166 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:33.598133    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15e0ff67-313d-4bad-93f1-c5924ea94c41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d39ad088-f9dfn\" (UID: \"15e0ff67-313d-4bad-93f1-c5924ea94c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d39ad088-f9dfn"
Oct 30 21:50:33.598317 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:33.598219    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpkq7\" (UniqueName: \"kubernetes.io/projected/15e0ff67-313d-4bad-93f1-c5924ea94c41-kube-api-access-rpkq7\") pod \"maas-api-revoke-sub-d39ad088-f9dfn\" (UID: \"15e0ff67-313d-4bad-93f1-c5924ea94c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d39ad088-f9dfn"
Oct 30 21:50:33.598443 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:33.598421    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15e0ff67-313d-4bad-93f1-c5924ea94c41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d39ad088-f9dfn\" (UID: \"15e0ff67-313d-4bad-93f1-c5924ea94c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d39ad088-f9dfn"
Oct 30 21:50:33.636223 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:33.636200    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:50:33.663280 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:33.663248    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d39ad088-f9dfn"]
Oct 30 21:50:33.699307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:33.699275    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15e0ff67-313d-4bad-93f1-c5924ea94c41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d39ad088-f9dfn\" (UID: \"15e0ff67-313d-4bad-93f1-c5924ea94c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d39ad088-f9dfn"
Oct 30 21:50:33.699521 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:33.699327    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpkq7\" (UniqueName: \"kubernetes.io/projected/15e0ff67-313d-4bad-93f1-c5924ea94c41-kube-api-access-rpkq7\") pod \"maas-api-revoke-sub-d39ad088-f9dfn\" (UID: \"15e0ff67-313d-4bad-93f1-c5924ea94c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d39ad088-f9dfn"
Oct 30 21:50:33.699521 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:33.699398    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15e0ff67-313d-4bad-93f1-c5924ea94c41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d39ad088-f9dfn\" (UID: \"15e0ff67-313d-4bad-93f1-c5924ea94c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d39ad088-f9dfn"
Oct 30 21:50:33.700012 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:33.699993    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15e0ff67-313d-4bad-93f1-c5924ea94c41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d39ad088-f9dfn\" (UID: \"15e0ff67-313d-4bad-93f1-c5924ea94c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d39ad088-f9dfn"
Oct 30 21:50:33.702156 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:33.702137    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15e0ff67-313d-4bad-93f1-c5924ea94c41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d39ad088-f9dfn\" (UID: \"15e0ff67-313d-4bad-93f1-c5924ea94c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d39ad088-f9dfn"
Oct 30 21:50:33.767401 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:33.763726    5087 patch_prober.go:29] interesting pod/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 21:50:33.767401 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:33.763798    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj" podUID="284b12e8-9e86-4406-8494-360e02159b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 21:50:33.767401 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:33.764839    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpkq7\" (UniqueName: \"kubernetes.io/projected/15e0ff67-313d-4bad-93f1-c5924ea94c41-kube-api-access-rpkq7\") pod \"maas-api-revoke-sub-d39ad088-f9dfn\" (UID: \"15e0ff67-313d-4bad-93f1-c5924ea94c41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d39ad088-f9dfn"
Oct 30 21:50:33.871441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:33.871334    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d39ad088-f9dfn"
Oct 30 21:50:34.276097 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:34.276071    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d39ad088-f9dfn"]
Oct 30 21:50:34.277881 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:50:34.277855    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e0ff67_313d_4bad_93f1_c5924ea94c41.slice/crio-0d76ef3bceef73baa91cd321d77974cdaf00e95fe290ce1a149049146ab80555 WatchSource:0}: Error finding container 0d76ef3bceef73baa91cd321d77974cdaf00e95fe290ce1a149049146ab80555: Status 404 returned error can't find the container with id 0d76ef3bceef73baa91cd321d77974cdaf00e95fe290ce1a149049146ab80555
Oct 30 21:50:34.762958 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:34.762728    5087 patch_prober.go:29] interesting pod/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 30 21:50:34.762958 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:34.762799    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj" podUID="284b12e8-9e86-4406-8494-360e02159b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 30 21:50:34.947428 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:34.946987    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d39ad088-f9dfn" event={"ID":"15e0ff67-313d-4bad-93f1-c5924ea94c41","Type":"ContainerStarted","Data":"00eb7a48a5a4d2b8fee65f8009228b538dd21739dde7866666a150fbeec2f00c"}
Oct 30 21:50:34.947428 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:34.947073    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d39ad088-f9dfn" event={"ID":"15e0ff67-313d-4bad-93f1-c5924ea94c41","Type":"ContainerStarted","Data":"0d76ef3bceef73baa91cd321d77974cdaf00e95fe290ce1a149049146ab80555"}
Oct 30 21:50:35.117888 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:35.117822    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d39ad088-f9dfn" podStartSLOduration=2.117803057 podStartE2EDuration="2.117803057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:35.066971764 +0000 UTC m=+1029.541475744" watchObservedRunningTime="2026-10-30 21:50:35.117803057 +0000 UTC m=+1029.592307038"
Oct 30 21:50:35.935328 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:35.935294    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:35.935718 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:35.935693    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:35.936552 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:35.936535    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 21:50:35.954938 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:35.954904    5087 generic.go:356] "Generic (PLEG): container finished" podID="15e0ff67-313d-4bad-93f1-c5924ea94c41" containerID="00eb7a48a5a4d2b8fee65f8009228b538dd21739dde7866666a150fbeec2f00c" exitCode=0
Oct 30 21:50:35.955308 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:35.954985    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d39ad088-f9dfn" event={"ID":"15e0ff67-313d-4bad-93f1-c5924ea94c41","Type":"ContainerDied","Data":"00eb7a48a5a4d2b8fee65f8009228b538dd21739dde7866666a150fbeec2f00c"}
Oct 30 21:50:37.112118 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:37.112095    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d39ad088-f9dfn"
Oct 30 21:50:37.135249 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:37.135227    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15e0ff67-313d-4bad-93f1-c5924ea94c41-kube-api-access-rpkq7\" (UniqueName: \"kubernetes.io/projected/15e0ff67-313d-4bad-93f1-c5924ea94c41-kube-api-access-rpkq7\") pod \"15e0ff67-313d-4bad-93f1-c5924ea94c41\" (UID: \"15e0ff67-313d-4bad-93f1-c5924ea94c41\") "
Oct 30 21:50:37.135385 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:37.135309    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15e0ff67-313d-4bad-93f1-c5924ea94c41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15e0ff67-313d-4bad-93f1-c5924ea94c41-maas-api-xks-ca\") pod \"15e0ff67-313d-4bad-93f1-c5924ea94c41\" (UID: \"15e0ff67-313d-4bad-93f1-c5924ea94c41\") "
Oct 30 21:50:37.135385 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:37.135359    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15e0ff67-313d-4bad-93f1-c5924ea94c41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15e0ff67-313d-4bad-93f1-c5924ea94c41-maas-api-openshift-ca\") pod \"15e0ff67-313d-4bad-93f1-c5924ea94c41\" (UID: \"15e0ff67-313d-4bad-93f1-c5924ea94c41\") "
Oct 30 21:50:37.135758 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:37.135731    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15e0ff67-313d-4bad-93f1-c5924ea94c41-maas-api-openshift-ca" pod "15e0ff67-313d-4bad-93f1-c5924ea94c41" (UID: "15e0ff67-313d-4bad-93f1-c5924ea94c41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:37.137406 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:37.137380    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e0ff67-313d-4bad-93f1-c5924ea94c41-kube-api-access-rpkq7" pod "15e0ff67-313d-4bad-93f1-c5924ea94c41" (UID: "15e0ff67-313d-4bad-93f1-c5924ea94c41"). InnerVolumeSpecName "kube-api-access-rpkq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:37.137513 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:37.137498    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15e0ff67-313d-4bad-93f1-c5924ea94c41-maas-api-xks-ca" pod "15e0ff67-313d-4bad-93f1-c5924ea94c41" (UID: "15e0ff67-313d-4bad-93f1-c5924ea94c41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:37.236283 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:37.236250    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpkq7\" (UniqueName: \"kubernetes.io/projected/15e0ff67-313d-4bad-93f1-c5924ea94c41-kube-api-access-rpkq7\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:37.236283 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:37.236282    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15e0ff67-313d-4bad-93f1-c5924ea94c41-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:37.236496 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:37.236296    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15e0ff67-313d-4bad-93f1-c5924ea94c41-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:37.278440 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:37.278403    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dec2888-nkdmc"]
Oct 30 21:50:37.279149 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:37.279133    5087 cpu_manager.go:414] "containerMap: removing container" podUID="15e0ff67-313d-4bad-93f1-c5924ea94c41" containerName="revoke-keys"
Oct 30 21:50:37.279191 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:37.279152    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15e0ff67-313d-4bad-93f1-c5924ea94c41" containerName="revoke-keys"
Oct 30 21:50:37.279364 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:37.279335    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="15e0ff67-313d-4bad-93f1-c5924ea94c41" containerName="revoke-keys"
Oct 30 21:50:37.283158 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:37.283136    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dec2888-nkdmc"
Oct 30 21:50:37.337768 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:37.337734    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zf9h\" (UniqueName: \"kubernetes.io/projected/af1c71a9-402d-4ac8-9cfa-a1ddf45f1274-kube-api-access-7zf9h\") pod \"maas-api-revoke-sub-9dec2888-nkdmc\" (UID: \"af1c71a9-402d-4ac8-9cfa-a1ddf45f1274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dec2888-nkdmc"
Oct 30 21:50:37.337930 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:37.337794    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1c71a9-402d-4ac8-9cfa-a1ddf45f1274-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dec2888-nkdmc\" (UID: \"af1c71a9-402d-4ac8-9cfa-a1ddf45f1274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dec2888-nkdmc"
Oct 30 21:50:37.337930 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:37.337898    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1c71a9-402d-4ac8-9cfa-a1ddf45f1274-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dec2888-nkdmc\" (UID: \"af1c71a9-402d-4ac8-9cfa-a1ddf45f1274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dec2888-nkdmc"
Oct 30 21:50:37.338982 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:37.338957    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dec2888-nkdmc"]
Oct 30 21:50:37.438507 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:37.438477    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zf9h\" (UniqueName: \"kubernetes.io/projected/af1c71a9-402d-4ac8-9cfa-a1ddf45f1274-kube-api-access-7zf9h\") pod \"maas-api-revoke-sub-9dec2888-nkdmc\" (UID: \"af1c71a9-402d-4ac8-9cfa-a1ddf45f1274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dec2888-nkdmc"
Oct 30 21:50:37.438649 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:37.438526    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1c71a9-402d-4ac8-9cfa-a1ddf45f1274-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dec2888-nkdmc\" (UID: \"af1c71a9-402d-4ac8-9cfa-a1ddf45f1274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dec2888-nkdmc"
Oct 30 21:50:37.438716 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:37.438693    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1c71a9-402d-4ac8-9cfa-a1ddf45f1274-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dec2888-nkdmc\" (UID: \"af1c71a9-402d-4ac8-9cfa-a1ddf45f1274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dec2888-nkdmc"
Oct 30 21:50:37.439429 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:37.439315    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1c71a9-402d-4ac8-9cfa-a1ddf45f1274-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dec2888-nkdmc\" (UID: \"af1c71a9-402d-4ac8-9cfa-a1ddf45f1274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dec2888-nkdmc"
Oct 30 21:50:37.441038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:37.441015    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1c71a9-402d-4ac8-9cfa-a1ddf45f1274-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dec2888-nkdmc\" (UID: \"af1c71a9-402d-4ac8-9cfa-a1ddf45f1274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dec2888-nkdmc"
Oct 30 21:50:37.513080 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:37.513018    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zf9h\" (UniqueName: \"kubernetes.io/projected/af1c71a9-402d-4ac8-9cfa-a1ddf45f1274-kube-api-access-7zf9h\") pod \"maas-api-revoke-sub-9dec2888-nkdmc\" (UID: \"af1c71a9-402d-4ac8-9cfa-a1ddf45f1274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dec2888-nkdmc"
Oct 30 21:50:37.596924 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:37.596897    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dec2888-nkdmc"
Oct 30 21:50:37.967614 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:37.967529    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d39ad088-f9dfn" event={"ID":"15e0ff67-313d-4bad-93f1-c5924ea94c41","Type":"ContainerDied","Data":"0d76ef3bceef73baa91cd321d77974cdaf00e95fe290ce1a149049146ab80555"}
Oct 30 21:50:37.967614 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:37.967569    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d39ad088-f9dfn"
Oct 30 21:50:37.967614 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:37.967583    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d76ef3bceef73baa91cd321d77974cdaf00e95fe290ce1a149049146ab80555"
Oct 30 21:50:37.973167 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:37.973143    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dec2888-nkdmc"]
Oct 30 21:50:37.973304 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:50:37.973284    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf1c71a9_402d_4ac8_9cfa_a1ddf45f1274.slice/crio-a53a5dbfe6df7f96524ec4a8752ee12906ee49dede6ade54c36f39de81016e67 WatchSource:0}: Error finding container a53a5dbfe6df7f96524ec4a8752ee12906ee49dede6ade54c36f39de81016e67: Status 404 returned error can't find the container with id a53a5dbfe6df7f96524ec4a8752ee12906ee49dede6ade54c36f39de81016e67
Oct 30 21:50:38.974918 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:38.974883    5087 generic.go:356] "Generic (PLEG): container finished" podID="af1c71a9-402d-4ac8-9cfa-a1ddf45f1274" containerID="700236ccc2d49227d83211fba18cb2b2ec3caf35823ede2b9fdb2255590fc603" exitCode=0
Oct 30 21:50:38.975318 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:38.974948    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dec2888-nkdmc" event={"ID":"af1c71a9-402d-4ac8-9cfa-a1ddf45f1274","Type":"ContainerDied","Data":"700236ccc2d49227d83211fba18cb2b2ec3caf35823ede2b9fdb2255590fc603"}
Oct 30 21:50:38.975318 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:38.975012    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dec2888-nkdmc" event={"ID":"af1c71a9-402d-4ac8-9cfa-a1ddf45f1274","Type":"ContainerStarted","Data":"a53a5dbfe6df7f96524ec4a8752ee12906ee49dede6ade54c36f39de81016e67"}
Oct 30 21:50:40.139223 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:40.139199    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dec2888-nkdmc"
Oct 30 21:50:40.168719 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:40.168695    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af1c71a9-402d-4ac8-9cfa-a1ddf45f1274-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1c71a9-402d-4ac8-9cfa-a1ddf45f1274-maas-api-openshift-ca\") pod \"af1c71a9-402d-4ac8-9cfa-a1ddf45f1274\" (UID: \"af1c71a9-402d-4ac8-9cfa-a1ddf45f1274\") "
Oct 30 21:50:40.168893 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:40.168804    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af1c71a9-402d-4ac8-9cfa-a1ddf45f1274-kube-api-access-7zf9h\" (UniqueName: \"kubernetes.io/projected/af1c71a9-402d-4ac8-9cfa-a1ddf45f1274-kube-api-access-7zf9h\") pod \"af1c71a9-402d-4ac8-9cfa-a1ddf45f1274\" (UID: \"af1c71a9-402d-4ac8-9cfa-a1ddf45f1274\") "
Oct 30 21:50:40.168893 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:40.168878    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af1c71a9-402d-4ac8-9cfa-a1ddf45f1274-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1c71a9-402d-4ac8-9cfa-a1ddf45f1274-maas-api-xks-ca\") pod \"af1c71a9-402d-4ac8-9cfa-a1ddf45f1274\" (UID: \"af1c71a9-402d-4ac8-9cfa-a1ddf45f1274\") "
Oct 30 21:50:40.169132 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:40.169106    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af1c71a9-402d-4ac8-9cfa-a1ddf45f1274-maas-api-openshift-ca" pod "af1c71a9-402d-4ac8-9cfa-a1ddf45f1274" (UID: "af1c71a9-402d-4ac8-9cfa-a1ddf45f1274"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:40.169283 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:40.169267    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1c71a9-402d-4ac8-9cfa-a1ddf45f1274-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:40.171112 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:40.171084    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af1c71a9-402d-4ac8-9cfa-a1ddf45f1274-kube-api-access-7zf9h" pod "af1c71a9-402d-4ac8-9cfa-a1ddf45f1274" (UID: "af1c71a9-402d-4ac8-9cfa-a1ddf45f1274"). InnerVolumeSpecName "kube-api-access-7zf9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:40.171210 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:40.171123    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af1c71a9-402d-4ac8-9cfa-a1ddf45f1274-maas-api-xks-ca" pod "af1c71a9-402d-4ac8-9cfa-a1ddf45f1274" (UID: "af1c71a9-402d-4ac8-9cfa-a1ddf45f1274"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:40.270284 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:40.270209    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zf9h\" (UniqueName: \"kubernetes.io/projected/af1c71a9-402d-4ac8-9cfa-a1ddf45f1274-kube-api-access-7zf9h\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:40.270284 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:40.270248    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1c71a9-402d-4ac8-9cfa-a1ddf45f1274-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:40.988759 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:40.988716    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dec2888-nkdmc" event={"ID":"af1c71a9-402d-4ac8-9cfa-a1ddf45f1274","Type":"ContainerDied","Data":"a53a5dbfe6df7f96524ec4a8752ee12906ee49dede6ade54c36f39de81016e67"}
Oct 30 21:50:40.988759 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:40.988737    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dec2888-nkdmc"
Oct 30 21:50:40.988959 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:40.988773    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a53a5dbfe6df7f96524ec4a8752ee12906ee49dede6ade54c36f39de81016e67"
Oct 30 21:50:43.447511 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:43.447472    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c29418cd-x4x29"]
Oct 30 21:50:43.448199 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:43.447976    5087 cpu_manager.go:414] "containerMap: removing container" podUID="af1c71a9-402d-4ac8-9cfa-a1ddf45f1274" containerName="revoke-keys"
Oct 30 21:50:43.448199 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:43.447990    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af1c71a9-402d-4ac8-9cfa-a1ddf45f1274" containerName="revoke-keys"
Oct 30 21:50:43.448199 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:43.448069    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="af1c71a9-402d-4ac8-9cfa-a1ddf45f1274" containerName="revoke-keys"
Oct 30 21:50:43.451548 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:43.451525    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c29418cd-x4x29"
Oct 30 21:50:43.503275 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:43.503243    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjbpm\" (UniqueName: \"kubernetes.io/projected/65c43919-ddb4-413b-81aa-4c137227475f-kube-api-access-mjbpm\") pod \"maas-api-revoke-sub-c29418cd-x4x29\" (UID: \"65c43919-ddb4-413b-81aa-4c137227475f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c29418cd-x4x29"
Oct 30 21:50:43.503553 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:43.503520    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c43919-ddb4-413b-81aa-4c137227475f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c29418cd-x4x29\" (UID: \"65c43919-ddb4-413b-81aa-4c137227475f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c29418cd-x4x29"
Oct 30 21:50:43.503706 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:43.503589    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c43919-ddb4-413b-81aa-4c137227475f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c29418cd-x4x29\" (UID: \"65c43919-ddb4-413b-81aa-4c137227475f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c29418cd-x4x29"
Oct 30 21:50:43.553207 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:43.553177    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:50:43.553423 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:43.553316    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:50:43.556118 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:43.556092    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c29418cd-x4x29"]
Oct 30 21:50:43.604556 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:43.604527    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c43919-ddb4-413b-81aa-4c137227475f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c29418cd-x4x29\" (UID: \"65c43919-ddb4-413b-81aa-4c137227475f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c29418cd-x4x29"
Oct 30 21:50:43.604556 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:43.604559    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c43919-ddb4-413b-81aa-4c137227475f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c29418cd-x4x29\" (UID: \"65c43919-ddb4-413b-81aa-4c137227475f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c29418cd-x4x29"
Oct 30 21:50:43.604722 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:43.604597    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjbpm\" (UniqueName: \"kubernetes.io/projected/65c43919-ddb4-413b-81aa-4c137227475f-kube-api-access-mjbpm\") pod \"maas-api-revoke-sub-c29418cd-x4x29\" (UID: \"65c43919-ddb4-413b-81aa-4c137227475f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c29418cd-x4x29"
Oct 30 21:50:43.605146 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:43.605127    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c43919-ddb4-413b-81aa-4c137227475f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c29418cd-x4x29\" (UID: \"65c43919-ddb4-413b-81aa-4c137227475f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c29418cd-x4x29"
Oct 30 21:50:43.606975 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:43.606956    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c43919-ddb4-413b-81aa-4c137227475f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c29418cd-x4x29\" (UID: \"65c43919-ddb4-413b-81aa-4c137227475f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c29418cd-x4x29"
Oct 30 21:50:43.661598 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:43.661565    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjbpm\" (UniqueName: \"kubernetes.io/projected/65c43919-ddb4-413b-81aa-4c137227475f-kube-api-access-mjbpm\") pod \"maas-api-revoke-sub-c29418cd-x4x29\" (UID: \"65c43919-ddb4-413b-81aa-4c137227475f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c29418cd-x4x29"
Oct 30 21:50:43.765561 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:43.765535    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c29418cd-x4x29"
Oct 30 21:50:44.138636 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:44.138604    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c29418cd-x4x29"]
Oct 30 21:50:44.140144 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:50:44.140109    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65c43919_ddb4_413b_81aa_4c137227475f.slice/crio-da372d441bd739958c29ca72818ff27ef71ce6b7e54bd1b9b6396dc348ec622a WatchSource:0}: Error finding container da372d441bd739958c29ca72818ff27ef71ce6b7e54bd1b9b6396dc348ec622a: Status 404 returned error can't find the container with id da372d441bd739958c29ca72818ff27ef71ce6b7e54bd1b9b6396dc348ec622a
Oct 30 21:50:45.014775 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:45.014742    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c29418cd-x4x29" event={"ID":"65c43919-ddb4-413b-81aa-4c137227475f","Type":"ContainerStarted","Data":"f149155a7f8011fcc1b3c5eef4113f4d66a89abaeaf5fb1f359bc8fd3b8e0a8b"}
Oct 30 21:50:45.015120 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:45.014810    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c29418cd-x4x29" event={"ID":"65c43919-ddb4-413b-81aa-4c137227475f","Type":"ContainerStarted","Data":"da372d441bd739958c29ca72818ff27ef71ce6b7e54bd1b9b6396dc348ec622a"}
Oct 30 21:50:46.022318 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:46.022286    5087 generic.go:356] "Generic (PLEG): container finished" podID="65c43919-ddb4-413b-81aa-4c137227475f" containerID="f149155a7f8011fcc1b3c5eef4113f4d66a89abaeaf5fb1f359bc8fd3b8e0a8b" exitCode=0
Oct 30 21:50:46.022318 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:46.022323    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c29418cd-x4x29" event={"ID":"65c43919-ddb4-413b-81aa-4c137227475f","Type":"ContainerDied","Data":"f149155a7f8011fcc1b3c5eef4113f4d66a89abaeaf5fb1f359bc8fd3b8e0a8b"}
Oct 30 21:50:46.170647 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:46.170622    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c29418cd-x4x29"
Oct 30 21:50:46.233092 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:46.233065    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65c43919-ddb4-413b-81aa-4c137227475f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c43919-ddb4-413b-81aa-4c137227475f-maas-api-xks-ca\") pod \"65c43919-ddb4-413b-81aa-4c137227475f\" (UID: \"65c43919-ddb4-413b-81aa-4c137227475f\") "
Oct 30 21:50:46.233264 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:46.233123    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65c43919-ddb4-413b-81aa-4c137227475f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c43919-ddb4-413b-81aa-4c137227475f-maas-api-openshift-ca\") pod \"65c43919-ddb4-413b-81aa-4c137227475f\" (UID: \"65c43919-ddb4-413b-81aa-4c137227475f\") "
Oct 30 21:50:46.233264 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:46.233165    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65c43919-ddb4-413b-81aa-4c137227475f-kube-api-access-mjbpm\" (UniqueName: \"kubernetes.io/projected/65c43919-ddb4-413b-81aa-4c137227475f-kube-api-access-mjbpm\") pod \"65c43919-ddb4-413b-81aa-4c137227475f\" (UID: \"65c43919-ddb4-413b-81aa-4c137227475f\") "
Oct 30 21:50:46.233539 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:46.233512    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65c43919-ddb4-413b-81aa-4c137227475f-maas-api-openshift-ca" pod "65c43919-ddb4-413b-81aa-4c137227475f" (UID: "65c43919-ddb4-413b-81aa-4c137227475f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:46.235338 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:46.235316    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65c43919-ddb4-413b-81aa-4c137227475f-kube-api-access-mjbpm" pod "65c43919-ddb4-413b-81aa-4c137227475f" (UID: "65c43919-ddb4-413b-81aa-4c137227475f"). InnerVolumeSpecName "kube-api-access-mjbpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:46.235466 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:46.235439    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65c43919-ddb4-413b-81aa-4c137227475f-maas-api-xks-ca" pod "65c43919-ddb4-413b-81aa-4c137227475f" (UID: "65c43919-ddb4-413b-81aa-4c137227475f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:46.334205 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:46.334135    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65c43919-ddb4-413b-81aa-4c137227475f-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:46.334205 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:46.334163    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65c43919-ddb4-413b-81aa-4c137227475f-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:46.334205 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:46.334172    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjbpm\" (UniqueName: \"kubernetes.io/projected/65c43919-ddb4-413b-81aa-4c137227475f-kube-api-access-mjbpm\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:47.028937 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:47.028909    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c29418cd-x4x29"
Oct 30 21:50:47.029330 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:47.028934    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c29418cd-x4x29" event={"ID":"65c43919-ddb4-413b-81aa-4c137227475f","Type":"ContainerDied","Data":"da372d441bd739958c29ca72818ff27ef71ce6b7e54bd1b9b6396dc348ec622a"}
Oct 30 21:50:47.029330 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:47.028986    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da372d441bd739958c29ca72818ff27ef71ce6b7e54bd1b9b6396dc348ec622a"
Oct 30 21:50:47.167006 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:50:47.166970    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:50:48.050906 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:48.050876    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9083dfa4-8t5bs"]
Oct 30 21:50:48.051464 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:48.051436    5087 cpu_manager.go:414] "containerMap: removing container" podUID="65c43919-ddb4-413b-81aa-4c137227475f" containerName="revoke-keys"
Oct 30 21:50:48.051544 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:48.051467    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65c43919-ddb4-413b-81aa-4c137227475f" containerName="revoke-keys"
Oct 30 21:50:48.051639 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:48.051627    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="65c43919-ddb4-413b-81aa-4c137227475f" containerName="revoke-keys"
Oct 30 21:50:48.055196 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:48.055174    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9083dfa4-8t5bs"
Oct 30 21:50:48.108395 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:48.108156    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:50:48.112193 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:48.112160    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9083dfa4-8t5bs"]
Oct 30 21:50:48.154715 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:48.154671    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6qlj\" (UniqueName: \"kubernetes.io/projected/8fdca4be-6f86-45e3-bbd3-3e78a283a6f5-kube-api-access-q6qlj\") pod \"maas-api-revoke-sub-9083dfa4-8t5bs\" (UID: \"8fdca4be-6f86-45e3-bbd3-3e78a283a6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9083dfa4-8t5bs"
Oct 30 21:50:48.154866 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:48.154829    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fdca4be-6f86-45e3-bbd3-3e78a283a6f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9083dfa4-8t5bs\" (UID: \"8fdca4be-6f86-45e3-bbd3-3e78a283a6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9083dfa4-8t5bs"
Oct 30 21:50:48.154931 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:48.154896    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fdca4be-6f86-45e3-bbd3-3e78a283a6f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9083dfa4-8t5bs\" (UID: \"8fdca4be-6f86-45e3-bbd3-3e78a283a6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9083dfa4-8t5bs"
Oct 30 21:50:48.157388 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:48.157371    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:50:48.256164 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:48.256136    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6qlj\" (UniqueName: \"kubernetes.io/projected/8fdca4be-6f86-45e3-bbd3-3e78a283a6f5-kube-api-access-q6qlj\") pod \"maas-api-revoke-sub-9083dfa4-8t5bs\" (UID: \"8fdca4be-6f86-45e3-bbd3-3e78a283a6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9083dfa4-8t5bs"
Oct 30 21:50:48.256329 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:48.256247    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fdca4be-6f86-45e3-bbd3-3e78a283a6f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9083dfa4-8t5bs\" (UID: \"8fdca4be-6f86-45e3-bbd3-3e78a283a6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9083dfa4-8t5bs"
Oct 30 21:50:48.256329 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:48.256313    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fdca4be-6f86-45e3-bbd3-3e78a283a6f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9083dfa4-8t5bs\" (UID: \"8fdca4be-6f86-45e3-bbd3-3e78a283a6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9083dfa4-8t5bs"
Oct 30 21:50:48.256879 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:48.256856    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fdca4be-6f86-45e3-bbd3-3e78a283a6f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9083dfa4-8t5bs\" (UID: \"8fdca4be-6f86-45e3-bbd3-3e78a283a6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9083dfa4-8t5bs"
Oct 30 21:50:48.258757 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:48.258739    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fdca4be-6f86-45e3-bbd3-3e78a283a6f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9083dfa4-8t5bs\" (UID: \"8fdca4be-6f86-45e3-bbd3-3e78a283a6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9083dfa4-8t5bs"
Oct 30 21:50:48.313046 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:48.312989    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6qlj\" (UniqueName: \"kubernetes.io/projected/8fdca4be-6f86-45e3-bbd3-3e78a283a6f5-kube-api-access-q6qlj\") pod \"maas-api-revoke-sub-9083dfa4-8t5bs\" (UID: \"8fdca4be-6f86-45e3-bbd3-3e78a283a6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9083dfa4-8t5bs"
Oct 30 21:50:48.372222 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:48.372197    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9083dfa4-8t5bs"
Oct 30 21:50:48.745289 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:48.745256    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9083dfa4-8t5bs"]
Oct 30 21:50:48.746209 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:50:48.746185    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fdca4be_6f86_45e3_bbd3_3e78a283a6f5.slice/crio-c3ef006221b8b2437a2b59a7775a79102e341cc919353ab4d1e7201c3a50866b WatchSource:0}: Error finding container c3ef006221b8b2437a2b59a7775a79102e341cc919353ab4d1e7201c3a50866b: Status 404 returned error can't find the container with id c3ef006221b8b2437a2b59a7775a79102e341cc919353ab4d1e7201c3a50866b
Oct 30 21:50:49.041405 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:49.041338    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9083dfa4-8t5bs" event={"ID":"8fdca4be-6f86-45e3-bbd3-3e78a283a6f5","Type":"ContainerStarted","Data":"c3ef006221b8b2437a2b59a7775a79102e341cc919353ab4d1e7201c3a50866b"}
Oct 30 21:50:50.049044 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:50.049001    5087 generic.go:356] "Generic (PLEG): container finished" podID="8fdca4be-6f86-45e3-bbd3-3e78a283a6f5" containerID="e416afe14a767908949fef63a5a11dc75ad74d8a6391e2b318ed51e26a10dd8b" exitCode=0
Oct 30 21:50:50.049456 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:50.049086    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9083dfa4-8t5bs" event={"ID":"8fdca4be-6f86-45e3-bbd3-3e78a283a6f5","Type":"ContainerDied","Data":"e416afe14a767908949fef63a5a11dc75ad74d8a6391e2b318ed51e26a10dd8b"}
Oct 30 21:50:51.209245 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:51.209222    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9083dfa4-8t5bs"
Oct 30 21:50:51.289628 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:51.289606    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fdca4be-6f86-45e3-bbd3-3e78a283a6f5-kube-api-access-q6qlj\" (UniqueName: \"kubernetes.io/projected/8fdca4be-6f86-45e3-bbd3-3e78a283a6f5-kube-api-access-q6qlj\") pod \"8fdca4be-6f86-45e3-bbd3-3e78a283a6f5\" (UID: \"8fdca4be-6f86-45e3-bbd3-3e78a283a6f5\") "
Oct 30 21:50:51.289757 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:51.289702    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fdca4be-6f86-45e3-bbd3-3e78a283a6f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fdca4be-6f86-45e3-bbd3-3e78a283a6f5-maas-api-openshift-ca\") pod \"8fdca4be-6f86-45e3-bbd3-3e78a283a6f5\" (UID: \"8fdca4be-6f86-45e3-bbd3-3e78a283a6f5\") "
Oct 30 21:50:51.289757 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:51.289733    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fdca4be-6f86-45e3-bbd3-3e78a283a6f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fdca4be-6f86-45e3-bbd3-3e78a283a6f5-maas-api-xks-ca\") pod \"8fdca4be-6f86-45e3-bbd3-3e78a283a6f5\" (UID: \"8fdca4be-6f86-45e3-bbd3-3e78a283a6f5\") "
Oct 30 21:50:51.290052 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:51.290026    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fdca4be-6f86-45e3-bbd3-3e78a283a6f5-maas-api-openshift-ca" pod "8fdca4be-6f86-45e3-bbd3-3e78a283a6f5" (UID: "8fdca4be-6f86-45e3-bbd3-3e78a283a6f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:51.291820 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:51.291797    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fdca4be-6f86-45e3-bbd3-3e78a283a6f5-maas-api-xks-ca" pod "8fdca4be-6f86-45e3-bbd3-3e78a283a6f5" (UID: "8fdca4be-6f86-45e3-bbd3-3e78a283a6f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:51.291922 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:51.291827    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fdca4be-6f86-45e3-bbd3-3e78a283a6f5-kube-api-access-q6qlj" pod "8fdca4be-6f86-45e3-bbd3-3e78a283a6f5" (UID: "8fdca4be-6f86-45e3-bbd3-3e78a283a6f5"). InnerVolumeSpecName "kube-api-access-q6qlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:51.391153 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:51.391072    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6qlj\" (UniqueName: \"kubernetes.io/projected/8fdca4be-6f86-45e3-bbd3-3e78a283a6f5-kube-api-access-q6qlj\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:51.391153 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:51.391102    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fdca4be-6f86-45e3-bbd3-3e78a283a6f5-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:51.391153 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:51.391113    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fdca4be-6f86-45e3-bbd3-3e78a283a6f5-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:52.061838 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:52.061805    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9083dfa4-8t5bs"
Oct 30 21:50:52.062019 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:52.061805    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9083dfa4-8t5bs" event={"ID":"8fdca4be-6f86-45e3-bbd3-3e78a283a6f5","Type":"ContainerDied","Data":"c3ef006221b8b2437a2b59a7775a79102e341cc919353ab4d1e7201c3a50866b"}
Oct 30 21:50:52.062019 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:52.061949    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3ef006221b8b2437a2b59a7775a79102e341cc919353ab4d1e7201c3a50866b"
Oct 30 21:50:55.259428 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:55.259379    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8af1a54-vh75s"]
Oct 30 21:50:55.259936 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:55.259861    5087 cpu_manager.go:414] "containerMap: removing container" podUID="8fdca4be-6f86-45e3-bbd3-3e78a283a6f5" containerName="revoke-keys"
Oct 30 21:50:55.259936 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:55.259873    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fdca4be-6f86-45e3-bbd3-3e78a283a6f5" containerName="revoke-keys"
Oct 30 21:50:55.260046 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:55.259984    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fdca4be-6f86-45e3-bbd3-3e78a283a6f5" containerName="revoke-keys"
Oct 30 21:50:55.263448 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:55.263426    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8af1a54-vh75s"
Oct 30 21:50:55.338860 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:55.338831    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a84962-471b-4ce2-9561-ceb900285afd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8af1a54-vh75s\" (UID: \"77a84962-471b-4ce2-9561-ceb900285afd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8af1a54-vh75s"
Oct 30 21:50:55.339020 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:55.338995    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8hck\" (UniqueName: \"kubernetes.io/projected/77a84962-471b-4ce2-9561-ceb900285afd-kube-api-access-c8hck\") pod \"maas-api-revoke-sub-d8af1a54-vh75s\" (UID: \"77a84962-471b-4ce2-9561-ceb900285afd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8af1a54-vh75s"
Oct 30 21:50:55.339095 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:55.339074    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a84962-471b-4ce2-9561-ceb900285afd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8af1a54-vh75s\" (UID: \"77a84962-471b-4ce2-9561-ceb900285afd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8af1a54-vh75s"
Oct 30 21:50:55.364577 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:55.364543    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:50:55.364751 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:55.364619    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:50:55.367797 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:55.367770    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8af1a54-vh75s"]
Oct 30 21:50:55.440624 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:55.440596    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8hck\" (UniqueName: \"kubernetes.io/projected/77a84962-471b-4ce2-9561-ceb900285afd-kube-api-access-c8hck\") pod \"maas-api-revoke-sub-d8af1a54-vh75s\" (UID: \"77a84962-471b-4ce2-9561-ceb900285afd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8af1a54-vh75s"
Oct 30 21:50:55.440809 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:55.440637    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a84962-471b-4ce2-9561-ceb900285afd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8af1a54-vh75s\" (UID: \"77a84962-471b-4ce2-9561-ceb900285afd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8af1a54-vh75s"
Oct 30 21:50:55.440809 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:55.440719    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a84962-471b-4ce2-9561-ceb900285afd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8af1a54-vh75s\" (UID: \"77a84962-471b-4ce2-9561-ceb900285afd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8af1a54-vh75s"
Oct 30 21:50:55.441640 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:55.441614    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a84962-471b-4ce2-9561-ceb900285afd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8af1a54-vh75s\" (UID: \"77a84962-471b-4ce2-9561-ceb900285afd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8af1a54-vh75s"
Oct 30 21:50:55.443486 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:55.443460    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a84962-471b-4ce2-9561-ceb900285afd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8af1a54-vh75s\" (UID: \"77a84962-471b-4ce2-9561-ceb900285afd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8af1a54-vh75s"
Oct 30 21:50:55.497841 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:55.497814    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8hck\" (UniqueName: \"kubernetes.io/projected/77a84962-471b-4ce2-9561-ceb900285afd-kube-api-access-c8hck\") pod \"maas-api-revoke-sub-d8af1a54-vh75s\" (UID: \"77a84962-471b-4ce2-9561-ceb900285afd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8af1a54-vh75s"
Oct 30 21:50:55.577864 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:55.577792    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8af1a54-vh75s"
Oct 30 21:50:55.955164 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:55.955135    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8af1a54-vh75s"]
Oct 30 21:50:55.957175 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:50:55.957144    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77a84962_471b_4ce2_9561_ceb900285afd.slice/crio-baaa7c447e89d2d9fdf26f6537d3c2fa6abce6e8337468775b08f082de998d76 WatchSource:0}: Error finding container baaa7c447e89d2d9fdf26f6537d3c2fa6abce6e8337468775b08f082de998d76: Status 404 returned error can't find the container with id baaa7c447e89d2d9fdf26f6537d3c2fa6abce6e8337468775b08f082de998d76
Oct 30 21:50:56.088217 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:56.088185    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8af1a54-vh75s" event={"ID":"77a84962-471b-4ce2-9561-ceb900285afd","Type":"ContainerStarted","Data":"baaa7c447e89d2d9fdf26f6537d3c2fa6abce6e8337468775b08f082de998d76"}
Oct 30 21:50:57.096179 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:57.096093    5087 generic.go:356] "Generic (PLEG): container finished" podID="77a84962-471b-4ce2-9561-ceb900285afd" containerID="daecc8c78236951ea4adf6d95ff7e0677cf5d6c30a103550aa9b914e9f20adb4" exitCode=0
Oct 30 21:50:57.096581 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:57.096178    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8af1a54-vh75s" event={"ID":"77a84962-471b-4ce2-9561-ceb900285afd","Type":"ContainerDied","Data":"daecc8c78236951ea4adf6d95ff7e0677cf5d6c30a103550aa9b914e9f20adb4"}
Oct 30 21:50:58.305317 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:58.305295    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8af1a54-vh75s"
Oct 30 21:50:58.477975 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:58.477886    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77a84962-471b-4ce2-9561-ceb900285afd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a84962-471b-4ce2-9561-ceb900285afd-maas-api-openshift-ca\") pod \"77a84962-471b-4ce2-9561-ceb900285afd\" (UID: \"77a84962-471b-4ce2-9561-ceb900285afd\") "
Oct 30 21:50:58.478144 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:58.478044    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77a84962-471b-4ce2-9561-ceb900285afd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a84962-471b-4ce2-9561-ceb900285afd-maas-api-xks-ca\") pod \"77a84962-471b-4ce2-9561-ceb900285afd\" (UID: \"77a84962-471b-4ce2-9561-ceb900285afd\") "
Oct 30 21:50:58.478144 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:58.478095    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77a84962-471b-4ce2-9561-ceb900285afd-kube-api-access-c8hck\" (UniqueName: \"kubernetes.io/projected/77a84962-471b-4ce2-9561-ceb900285afd-kube-api-access-c8hck\") pod \"77a84962-471b-4ce2-9561-ceb900285afd\" (UID: \"77a84962-471b-4ce2-9561-ceb900285afd\") "
Oct 30 21:50:58.478394 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:58.478333    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77a84962-471b-4ce2-9561-ceb900285afd-maas-api-openshift-ca" pod "77a84962-471b-4ce2-9561-ceb900285afd" (UID: "77a84962-471b-4ce2-9561-ceb900285afd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:58.478526 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:58.478507    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77a84962-471b-4ce2-9561-ceb900285afd-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:58.480315 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:58.480287    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77a84962-471b-4ce2-9561-ceb900285afd-maas-api-xks-ca" pod "77a84962-471b-4ce2-9561-ceb900285afd" (UID: "77a84962-471b-4ce2-9561-ceb900285afd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:58.480476 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:58.480455    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77a84962-471b-4ce2-9561-ceb900285afd-kube-api-access-c8hck" pod "77a84962-471b-4ce2-9561-ceb900285afd" (UID: "77a84962-471b-4ce2-9561-ceb900285afd"). InnerVolumeSpecName "kube-api-access-c8hck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:58.579636 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:58.579599    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77a84962-471b-4ce2-9561-ceb900285afd-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:58.579636 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:58.579625    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8hck\" (UniqueName: \"kubernetes.io/projected/77a84962-471b-4ce2-9561-ceb900285afd-kube-api-access-c8hck\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:50:59.110901 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:59.110873    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8af1a54-vh75s"
Oct 30 21:50:59.111074 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:59.110912    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8af1a54-vh75s" event={"ID":"77a84962-471b-4ce2-9561-ceb900285afd","Type":"ContainerDied","Data":"baaa7c447e89d2d9fdf26f6537d3c2fa6abce6e8337468775b08f082de998d76"}
Oct 30 21:50:59.111074 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:59.110976    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="baaa7c447e89d2d9fdf26f6537d3c2fa6abce6e8337468775b08f082de998d76"
Oct 30 21:50:59.662124 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:59.662092    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3ca8f96-j62t6"]
Oct 30 21:50:59.662589 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:59.662575    5087 cpu_manager.go:414] "containerMap: removing container" podUID="77a84962-471b-4ce2-9561-ceb900285afd" containerName="revoke-keys"
Oct 30 21:50:59.662628 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:59.662590    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77a84962-471b-4ce2-9561-ceb900285afd" containerName="revoke-keys"
Oct 30 21:50:59.662700 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:59.662690    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="77a84962-471b-4ce2-9561-ceb900285afd" containerName="revoke-keys"
Oct 30 21:50:59.666139 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:59.666119    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ca8f96-j62t6"
Oct 30 21:50:59.719265 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:59.719223    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:50:59.719736 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:59.719712    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:50:59.723144 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:59.723103    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3ca8f96-j62t6"]
Oct 30 21:50:59.794011 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:59.793973    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfee1062-17e6-4d74-9140-f4eed1f21a07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3ca8f96-j62t6\" (UID: \"bfee1062-17e6-4d74-9140-f4eed1f21a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ca8f96-j62t6"
Oct 30 21:50:59.794225 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:59.794121    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfee1062-17e6-4d74-9140-f4eed1f21a07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3ca8f96-j62t6\" (UID: \"bfee1062-17e6-4d74-9140-f4eed1f21a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ca8f96-j62t6"
Oct 30 21:50:59.794225 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:59.794185    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njf4f\" (UniqueName: \"kubernetes.io/projected/bfee1062-17e6-4d74-9140-f4eed1f21a07-kube-api-access-njf4f\") pod \"maas-api-revoke-sub-f3ca8f96-j62t6\" (UID: \"bfee1062-17e6-4d74-9140-f4eed1f21a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ca8f96-j62t6"
Oct 30 21:50:59.894824 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:59.894796    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfee1062-17e6-4d74-9140-f4eed1f21a07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3ca8f96-j62t6\" (UID: \"bfee1062-17e6-4d74-9140-f4eed1f21a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ca8f96-j62t6"
Oct 30 21:50:59.895006 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:59.894878    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfee1062-17e6-4d74-9140-f4eed1f21a07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3ca8f96-j62t6\" (UID: \"bfee1062-17e6-4d74-9140-f4eed1f21a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ca8f96-j62t6"
Oct 30 21:50:59.895006 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:59.894926    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njf4f\" (UniqueName: \"kubernetes.io/projected/bfee1062-17e6-4d74-9140-f4eed1f21a07-kube-api-access-njf4f\") pod \"maas-api-revoke-sub-f3ca8f96-j62t6\" (UID: \"bfee1062-17e6-4d74-9140-f4eed1f21a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ca8f96-j62t6"
Oct 30 21:50:59.895566 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:59.895540    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfee1062-17e6-4d74-9140-f4eed1f21a07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3ca8f96-j62t6\" (UID: \"bfee1062-17e6-4d74-9140-f4eed1f21a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ca8f96-j62t6"
Oct 30 21:50:59.897509 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:59.897489    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfee1062-17e6-4d74-9140-f4eed1f21a07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3ca8f96-j62t6\" (UID: \"bfee1062-17e6-4d74-9140-f4eed1f21a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ca8f96-j62t6"
Oct 30 21:50:59.955680 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:59.955615    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njf4f\" (UniqueName: \"kubernetes.io/projected/bfee1062-17e6-4d74-9140-f4eed1f21a07-kube-api-access-njf4f\") pod \"maas-api-revoke-sub-f3ca8f96-j62t6\" (UID: \"bfee1062-17e6-4d74-9140-f4eed1f21a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ca8f96-j62t6"
Oct 30 21:50:59.978918 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:50:59.978894    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ca8f96-j62t6"
Oct 30 21:51:00.358193 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:00.358164    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3ca8f96-j62t6"]
Oct 30 21:51:00.359329 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:51:00.359307    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfee1062_17e6_4d74_9140_f4eed1f21a07.slice/crio-d4b15078c3fb3a6fee302b0da548bd9352c24c68d40e82059cf9a9ce2689b9bd WatchSource:0}: Error finding container d4b15078c3fb3a6fee302b0da548bd9352c24c68d40e82059cf9a9ce2689b9bd: Status 404 returned error can't find the container with id d4b15078c3fb3a6fee302b0da548bd9352c24c68d40e82059cf9a9ce2689b9bd
Oct 30 21:51:01.129209 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:01.129175    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ca8f96-j62t6" event={"ID":"bfee1062-17e6-4d74-9140-f4eed1f21a07","Type":"ContainerStarted","Data":"50418632b93d4ba8db820f4ce87a9dce4db1577c82a41899a6aa09cd7f164caf"}
Oct 30 21:51:01.129649 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:01.129234    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ca8f96-j62t6" event={"ID":"bfee1062-17e6-4d74-9140-f4eed1f21a07","Type":"ContainerStarted","Data":"d4b15078c3fb3a6fee302b0da548bd9352c24c68d40e82059cf9a9ce2689b9bd"}
Oct 30 21:51:01.248383 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:01.247681    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ca8f96-j62t6" podStartSLOduration=2.247660168 podStartE2EDuration="2.247660168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:01.242821696 +0000 UTC m=+1055.717325681" watchObservedRunningTime="2026-10-30 21:51:01.247660168 +0000 UTC m=+1055.722164149"
Oct 30 21:51:02.136263 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:02.136224    5087 generic.go:356] "Generic (PLEG): container finished" podID="bfee1062-17e6-4d74-9140-f4eed1f21a07" containerID="50418632b93d4ba8db820f4ce87a9dce4db1577c82a41899a6aa09cd7f164caf" exitCode=0
Oct 30 21:51:02.136698 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:02.136284    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ca8f96-j62t6" event={"ID":"bfee1062-17e6-4d74-9140-f4eed1f21a07","Type":"ContainerDied","Data":"50418632b93d4ba8db820f4ce87a9dce4db1577c82a41899a6aa09cd7f164caf"}
Oct 30 21:51:03.289837 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:03.289814    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ca8f96-j62t6"
Oct 30 21:51:03.331646 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:03.331611    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfee1062-17e6-4d74-9140-f4eed1f21a07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfee1062-17e6-4d74-9140-f4eed1f21a07-maas-api-openshift-ca\") pod \"bfee1062-17e6-4d74-9140-f4eed1f21a07\" (UID: \"bfee1062-17e6-4d74-9140-f4eed1f21a07\") "
Oct 30 21:51:03.331815 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:03.331721    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfee1062-17e6-4d74-9140-f4eed1f21a07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfee1062-17e6-4d74-9140-f4eed1f21a07-maas-api-xks-ca\") pod \"bfee1062-17e6-4d74-9140-f4eed1f21a07\" (UID: \"bfee1062-17e6-4d74-9140-f4eed1f21a07\") "
Oct 30 21:51:03.331815 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:03.331762    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfee1062-17e6-4d74-9140-f4eed1f21a07-kube-api-access-njf4f\" (UniqueName: \"kubernetes.io/projected/bfee1062-17e6-4d74-9140-f4eed1f21a07-kube-api-access-njf4f\") pod \"bfee1062-17e6-4d74-9140-f4eed1f21a07\" (UID: \"bfee1062-17e6-4d74-9140-f4eed1f21a07\") "
Oct 30 21:51:03.332004 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:03.331979    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfee1062-17e6-4d74-9140-f4eed1f21a07-maas-api-openshift-ca" pod "bfee1062-17e6-4d74-9140-f4eed1f21a07" (UID: "bfee1062-17e6-4d74-9140-f4eed1f21a07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:51:03.334052 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:03.334023    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfee1062-17e6-4d74-9140-f4eed1f21a07-kube-api-access-njf4f" pod "bfee1062-17e6-4d74-9140-f4eed1f21a07" (UID: "bfee1062-17e6-4d74-9140-f4eed1f21a07"). InnerVolumeSpecName "kube-api-access-njf4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:51:03.334149 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:03.334124    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfee1062-17e6-4d74-9140-f4eed1f21a07-maas-api-xks-ca" pod "bfee1062-17e6-4d74-9140-f4eed1f21a07" (UID: "bfee1062-17e6-4d74-9140-f4eed1f21a07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:51:03.432956 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:03.432868    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfee1062-17e6-4d74-9140-f4eed1f21a07-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:03.432956 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:03.432895    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njf4f\" (UniqueName: \"kubernetes.io/projected/bfee1062-17e6-4d74-9140-f4eed1f21a07-kube-api-access-njf4f\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:03.432956 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:03.432907    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfee1062-17e6-4d74-9140-f4eed1f21a07-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:04.150049 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:04.150018    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ca8f96-j62t6"
Oct 30 21:51:04.150221 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:04.150018    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ca8f96-j62t6" event={"ID":"bfee1062-17e6-4d74-9140-f4eed1f21a07","Type":"ContainerDied","Data":"d4b15078c3fb3a6fee302b0da548bd9352c24c68d40e82059cf9a9ce2689b9bd"}
Oct 30 21:51:04.150221 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:04.150106    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4b15078c3fb3a6fee302b0da548bd9352c24c68d40e82059cf9a9ce2689b9bd"
Oct 30 21:51:04.500621 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:04.500591    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38fb985e-96797"]
Oct 30 21:51:04.501048 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:04.501034    5087 cpu_manager.go:414] "containerMap: removing container" podUID="bfee1062-17e6-4d74-9140-f4eed1f21a07" containerName="revoke-keys"
Oct 30 21:51:04.501100 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:04.501049    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfee1062-17e6-4d74-9140-f4eed1f21a07" containerName="revoke-keys"
Oct 30 21:51:04.501173 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:04.501164    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfee1062-17e6-4d74-9140-f4eed1f21a07" containerName="revoke-keys"
Oct 30 21:51:04.504504 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:04.504481    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38fb985e-96797"
Oct 30 21:51:04.540342 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:04.540316    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3bcf59-d6bb-4198-a397-8d4432360425-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38fb985e-96797\" (UID: \"9e3bcf59-d6bb-4198-a397-8d4432360425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38fb985e-96797"
Oct 30 21:51:04.540488 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:04.540402    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3bcf59-d6bb-4198-a397-8d4432360425-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38fb985e-96797\" (UID: \"9e3bcf59-d6bb-4198-a397-8d4432360425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38fb985e-96797"
Oct 30 21:51:04.540488 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:04.540435    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfrqs\" (UniqueName: \"kubernetes.io/projected/9e3bcf59-d6bb-4198-a397-8d4432360425-kube-api-access-qfrqs\") pod \"maas-api-revoke-sub-38fb985e-96797\" (UID: \"9e3bcf59-d6bb-4198-a397-8d4432360425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38fb985e-96797"
Oct 30 21:51:04.557670 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:04.557643    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:51:04.557670 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:04.557641    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:51:04.561529 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:04.561503    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38fb985e-96797"]
Oct 30 21:51:04.641438 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:04.641409    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3bcf59-d6bb-4198-a397-8d4432360425-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38fb985e-96797\" (UID: \"9e3bcf59-d6bb-4198-a397-8d4432360425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38fb985e-96797"
Oct 30 21:51:04.641597 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:04.641446    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3bcf59-d6bb-4198-a397-8d4432360425-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38fb985e-96797\" (UID: \"9e3bcf59-d6bb-4198-a397-8d4432360425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38fb985e-96797"
Oct 30 21:51:04.641597 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:04.641473    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfrqs\" (UniqueName: \"kubernetes.io/projected/9e3bcf59-d6bb-4198-a397-8d4432360425-kube-api-access-qfrqs\") pod \"maas-api-revoke-sub-38fb985e-96797\" (UID: \"9e3bcf59-d6bb-4198-a397-8d4432360425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38fb985e-96797"
Oct 30 21:51:04.642058 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:04.642027    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3bcf59-d6bb-4198-a397-8d4432360425-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38fb985e-96797\" (UID: \"9e3bcf59-d6bb-4198-a397-8d4432360425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38fb985e-96797"
Oct 30 21:51:04.643709 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:04.643679    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3bcf59-d6bb-4198-a397-8d4432360425-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38fb985e-96797\" (UID: \"9e3bcf59-d6bb-4198-a397-8d4432360425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38fb985e-96797"
Oct 30 21:51:04.697469 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:04.697446    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfrqs\" (UniqueName: \"kubernetes.io/projected/9e3bcf59-d6bb-4198-a397-8d4432360425-kube-api-access-qfrqs\") pod \"maas-api-revoke-sub-38fb985e-96797\" (UID: \"9e3bcf59-d6bb-4198-a397-8d4432360425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38fb985e-96797"
Oct 30 21:51:04.818136 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:04.818064    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38fb985e-96797"
Oct 30 21:51:05.195130 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:05.195105    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38fb985e-96797"]
Oct 30 21:51:05.196952 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:51:05.196925    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3bcf59_d6bb_4198_a397_8d4432360425.slice/crio-0ee25d14a2b39ea6283db868d0d7642bf8eb8c1d969a5c1c71dae665abafadc7 WatchSource:0}: Error finding container 0ee25d14a2b39ea6283db868d0d7642bf8eb8c1d969a5c1c71dae665abafadc7: Status 404 returned error can't find the container with id 0ee25d14a2b39ea6283db868d0d7642bf8eb8c1d969a5c1c71dae665abafadc7
Oct 30 21:51:06.108577 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:51:06.108536    5087 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3bcf59_d6bb_4198_a397_8d4432360425.slice/crio-f4ec1253e27a340249567ffa6d83b5e03e55068823837a1332dcd6007c6bc6c0.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3bcf59_d6bb_4198_a397_8d4432360425.slice/crio-conmon-f4ec1253e27a340249567ffa6d83b5e03e55068823837a1332dcd6007c6bc6c0.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 21:51:06.164500 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:06.164461    5087 generic.go:356] "Generic (PLEG): container finished" podID="9e3bcf59-d6bb-4198-a397-8d4432360425" containerID="f4ec1253e27a340249567ffa6d83b5e03e55068823837a1332dcd6007c6bc6c0" exitCode=0
Oct 30 21:51:06.164682 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:06.164551    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38fb985e-96797" event={"ID":"9e3bcf59-d6bb-4198-a397-8d4432360425","Type":"ContainerDied","Data":"f4ec1253e27a340249567ffa6d83b5e03e55068823837a1332dcd6007c6bc6c0"}
Oct 30 21:51:06.164682 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:06.164634    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38fb985e-96797" event={"ID":"9e3bcf59-d6bb-4198-a397-8d4432360425","Type":"ContainerStarted","Data":"0ee25d14a2b39ea6283db868d0d7642bf8eb8c1d969a5c1c71dae665abafadc7"}
Oct 30 21:51:07.324917 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:07.324894    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38fb985e-96797"
Oct 30 21:51:07.369245 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:07.369218    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e3bcf59-d6bb-4198-a397-8d4432360425-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3bcf59-d6bb-4198-a397-8d4432360425-maas-api-xks-ca\") pod \"9e3bcf59-d6bb-4198-a397-8d4432360425\" (UID: \"9e3bcf59-d6bb-4198-a397-8d4432360425\") "
Oct 30 21:51:07.369422 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:07.369291    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e3bcf59-d6bb-4198-a397-8d4432360425-kube-api-access-qfrqs\" (UniqueName: \"kubernetes.io/projected/9e3bcf59-d6bb-4198-a397-8d4432360425-kube-api-access-qfrqs\") pod \"9e3bcf59-d6bb-4198-a397-8d4432360425\" (UID: \"9e3bcf59-d6bb-4198-a397-8d4432360425\") "
Oct 30 21:51:07.369422 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:07.369394    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e3bcf59-d6bb-4198-a397-8d4432360425-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3bcf59-d6bb-4198-a397-8d4432360425-maas-api-openshift-ca\") pod \"9e3bcf59-d6bb-4198-a397-8d4432360425\" (UID: \"9e3bcf59-d6bb-4198-a397-8d4432360425\") "
Oct 30 21:51:07.369774 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:07.369745    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e3bcf59-d6bb-4198-a397-8d4432360425-maas-api-openshift-ca" pod "9e3bcf59-d6bb-4198-a397-8d4432360425" (UID: "9e3bcf59-d6bb-4198-a397-8d4432360425"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:51:07.371693 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:07.371659    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e3bcf59-d6bb-4198-a397-8d4432360425-maas-api-xks-ca" pod "9e3bcf59-d6bb-4198-a397-8d4432360425" (UID: "9e3bcf59-d6bb-4198-a397-8d4432360425"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:51:07.371927 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:07.371903    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3bcf59-d6bb-4198-a397-8d4432360425-kube-api-access-qfrqs" pod "9e3bcf59-d6bb-4198-a397-8d4432360425" (UID: "9e3bcf59-d6bb-4198-a397-8d4432360425"). InnerVolumeSpecName "kube-api-access-qfrqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:51:07.470612 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:07.470523    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3bcf59-d6bb-4198-a397-8d4432360425-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:07.470612 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:07.470555    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3bcf59-d6bb-4198-a397-8d4432360425-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:07.470612 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:07.470565    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfrqs\" (UniqueName: \"kubernetes.io/projected/9e3bcf59-d6bb-4198-a397-8d4432360425-kube-api-access-qfrqs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:07.490381 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:07.490328    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41cf4770-bq55n"]
Oct 30 21:51:07.490882 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:07.490840    5087 cpu_manager.go:414] "containerMap: removing container" podUID="9e3bcf59-d6bb-4198-a397-8d4432360425" containerName="revoke-keys"
Oct 30 21:51:07.490882 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:07.490856    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e3bcf59-d6bb-4198-a397-8d4432360425" containerName="revoke-keys"
Oct 30 21:51:07.490959 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:07.490936    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e3bcf59-d6bb-4198-a397-8d4432360425" containerName="revoke-keys"
Oct 30 21:51:07.494408 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:07.494391    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cf4770-bq55n"
Oct 30 21:51:07.551228 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:07.551185    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41cf4770-bq55n"]
Oct 30 21:51:07.571746 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:07.571714    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4640ac0e-2df6-42ca-8ff9-8b32d1e386fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41cf4770-bq55n\" (UID: \"4640ac0e-2df6-42ca-8ff9-8b32d1e386fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cf4770-bq55n"
Oct 30 21:51:07.571898 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:07.571778    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4640ac0e-2df6-42ca-8ff9-8b32d1e386fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41cf4770-bq55n\" (UID: \"4640ac0e-2df6-42ca-8ff9-8b32d1e386fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cf4770-bq55n"
Oct 30 21:51:07.571979 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:07.571929    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpw79\" (UniqueName: \"kubernetes.io/projected/4640ac0e-2df6-42ca-8ff9-8b32d1e386fa-kube-api-access-xpw79\") pod \"maas-api-revoke-sub-41cf4770-bq55n\" (UID: \"4640ac0e-2df6-42ca-8ff9-8b32d1e386fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cf4770-bq55n"
Oct 30 21:51:07.672981 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:07.672949    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpw79\" (UniqueName: \"kubernetes.io/projected/4640ac0e-2df6-42ca-8ff9-8b32d1e386fa-kube-api-access-xpw79\") pod \"maas-api-revoke-sub-41cf4770-bq55n\" (UID: \"4640ac0e-2df6-42ca-8ff9-8b32d1e386fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cf4770-bq55n"
Oct 30 21:51:07.673188 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:07.673083    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4640ac0e-2df6-42ca-8ff9-8b32d1e386fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41cf4770-bq55n\" (UID: \"4640ac0e-2df6-42ca-8ff9-8b32d1e386fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cf4770-bq55n"
Oct 30 21:51:07.673188 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:07.673145    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4640ac0e-2df6-42ca-8ff9-8b32d1e386fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41cf4770-bq55n\" (UID: \"4640ac0e-2df6-42ca-8ff9-8b32d1e386fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cf4770-bq55n"
Oct 30 21:51:07.673741 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:07.673716    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4640ac0e-2df6-42ca-8ff9-8b32d1e386fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41cf4770-bq55n\" (UID: \"4640ac0e-2df6-42ca-8ff9-8b32d1e386fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cf4770-bq55n"
Oct 30 21:51:07.675762 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:07.675734    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4640ac0e-2df6-42ca-8ff9-8b32d1e386fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41cf4770-bq55n\" (UID: \"4640ac0e-2df6-42ca-8ff9-8b32d1e386fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cf4770-bq55n"
Oct 30 21:51:07.731411 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:07.731331    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpw79\" (UniqueName: \"kubernetes.io/projected/4640ac0e-2df6-42ca-8ff9-8b32d1e386fa-kube-api-access-xpw79\") pod \"maas-api-revoke-sub-41cf4770-bq55n\" (UID: \"4640ac0e-2df6-42ca-8ff9-8b32d1e386fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cf4770-bq55n"
Oct 30 21:51:07.809390 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:07.809341    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cf4770-bq55n"
Oct 30 21:51:08.182040 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:08.181970    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38fb985e-96797"
Oct 30 21:51:08.182040 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:08.181998    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38fb985e-96797" event={"ID":"9e3bcf59-d6bb-4198-a397-8d4432360425","Type":"ContainerDied","Data":"0ee25d14a2b39ea6283db868d0d7642bf8eb8c1d969a5c1c71dae665abafadc7"}
Oct 30 21:51:08.182257 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:08.182042    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ee25d14a2b39ea6283db868d0d7642bf8eb8c1d969a5c1c71dae665abafadc7"
Oct 30 21:51:08.189811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:08.189783    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41cf4770-bq55n"]
Oct 30 21:51:08.194875 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:51:08.194829    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4640ac0e_2df6_42ca_8ff9_8b32d1e386fa.slice/crio-17b120a7e671ec5399ebe9320b9ceaad9dfe0faf28df01b4cc202116f533f748 WatchSource:0}: Error finding container 17b120a7e671ec5399ebe9320b9ceaad9dfe0faf28df01b4cc202116f533f748: Status 404 returned error can't find the container with id 17b120a7e671ec5399ebe9320b9ceaad9dfe0faf28df01b4cc202116f533f748
Oct 30 21:51:09.193588 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:09.193552    5087 generic.go:356] "Generic (PLEG): container finished" podID="4640ac0e-2df6-42ca-8ff9-8b32d1e386fa" containerID="3502f71574093a6a118ae4200d7d5080456f779a29b1250e4c51a5101db2d6d4" exitCode=0
Oct 30 21:51:09.194079 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:09.193664    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cf4770-bq55n" event={"ID":"4640ac0e-2df6-42ca-8ff9-8b32d1e386fa","Type":"ContainerDied","Data":"3502f71574093a6a118ae4200d7d5080456f779a29b1250e4c51a5101db2d6d4"}
Oct 30 21:51:09.194079 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:09.193718    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cf4770-bq55n" event={"ID":"4640ac0e-2df6-42ca-8ff9-8b32d1e386fa","Type":"ContainerStarted","Data":"17b120a7e671ec5399ebe9320b9ceaad9dfe0faf28df01b4cc202116f533f748"}
Oct 30 21:51:10.338607 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:10.338583    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cf4770-bq55n"
Oct 30 21:51:10.404020 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:10.403993    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4640ac0e-2df6-42ca-8ff9-8b32d1e386fa-kube-api-access-xpw79\" (UniqueName: \"kubernetes.io/projected/4640ac0e-2df6-42ca-8ff9-8b32d1e386fa-kube-api-access-xpw79\") pod \"4640ac0e-2df6-42ca-8ff9-8b32d1e386fa\" (UID: \"4640ac0e-2df6-42ca-8ff9-8b32d1e386fa\") "
Oct 30 21:51:10.404166 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:10.404035    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4640ac0e-2df6-42ca-8ff9-8b32d1e386fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4640ac0e-2df6-42ca-8ff9-8b32d1e386fa-maas-api-xks-ca\") pod \"4640ac0e-2df6-42ca-8ff9-8b32d1e386fa\" (UID: \"4640ac0e-2df6-42ca-8ff9-8b32d1e386fa\") "
Oct 30 21:51:10.404166 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:10.404083    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4640ac0e-2df6-42ca-8ff9-8b32d1e386fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4640ac0e-2df6-42ca-8ff9-8b32d1e386fa-maas-api-openshift-ca\") pod \"4640ac0e-2df6-42ca-8ff9-8b32d1e386fa\" (UID: \"4640ac0e-2df6-42ca-8ff9-8b32d1e386fa\") "
Oct 30 21:51:10.404552 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:10.404522    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4640ac0e-2df6-42ca-8ff9-8b32d1e386fa-maas-api-openshift-ca" pod "4640ac0e-2df6-42ca-8ff9-8b32d1e386fa" (UID: "4640ac0e-2df6-42ca-8ff9-8b32d1e386fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:51:10.404673 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:10.404620    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4640ac0e-2df6-42ca-8ff9-8b32d1e386fa-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:10.406127 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:10.406109    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4640ac0e-2df6-42ca-8ff9-8b32d1e386fa-kube-api-access-xpw79" pod "4640ac0e-2df6-42ca-8ff9-8b32d1e386fa" (UID: "4640ac0e-2df6-42ca-8ff9-8b32d1e386fa"). InnerVolumeSpecName "kube-api-access-xpw79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:51:10.406397 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:10.406321    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4640ac0e-2df6-42ca-8ff9-8b32d1e386fa-maas-api-xks-ca" pod "4640ac0e-2df6-42ca-8ff9-8b32d1e386fa" (UID: "4640ac0e-2df6-42ca-8ff9-8b32d1e386fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:51:10.505925 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:10.505894    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpw79\" (UniqueName: \"kubernetes.io/projected/4640ac0e-2df6-42ca-8ff9-8b32d1e386fa-kube-api-access-xpw79\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:10.505925 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:10.505921    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4640ac0e-2df6-42ca-8ff9-8b32d1e386fa-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:11.208556 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:11.208520    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cf4770-bq55n"
Oct 30 21:51:11.208556 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:11.208551    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41cf4770-bq55n" event={"ID":"4640ac0e-2df6-42ca-8ff9-8b32d1e386fa","Type":"ContainerDied","Data":"17b120a7e671ec5399ebe9320b9ceaad9dfe0faf28df01b4cc202116f533f748"}
Oct 30 21:51:11.208759 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:11.208603    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17b120a7e671ec5399ebe9320b9ceaad9dfe0faf28df01b4cc202116f533f748"
Oct 30 21:51:12.562301 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:12.562265    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18b65dbd-bxxqx"]
Oct 30 21:51:12.562808 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:12.562762    5087 cpu_manager.go:414] "containerMap: removing container" podUID="4640ac0e-2df6-42ca-8ff9-8b32d1e386fa" containerName="revoke-keys"
Oct 30 21:51:12.562808 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:12.562774    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4640ac0e-2df6-42ca-8ff9-8b32d1e386fa" containerName="revoke-keys"
Oct 30 21:51:12.562922 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:12.562880    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="4640ac0e-2df6-42ca-8ff9-8b32d1e386fa" containerName="revoke-keys"
Oct 30 21:51:12.566383 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:12.566363    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b65dbd-bxxqx"
Oct 30 21:51:12.620490 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:12.620439    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:51:12.627109 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:12.627078    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7171f5cd-a37b-499f-a06a-2a9b4a121da2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18b65dbd-bxxqx\" (UID: \"7171f5cd-a37b-499f-a06a-2a9b4a121da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b65dbd-bxxqx"
Oct 30 21:51:12.627285 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:12.627158    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7171f5cd-a37b-499f-a06a-2a9b4a121da2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18b65dbd-bxxqx\" (UID: \"7171f5cd-a37b-499f-a06a-2a9b4a121da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b65dbd-bxxqx"
Oct 30 21:51:12.627285 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:12.627243    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfllw\" (UniqueName: \"kubernetes.io/projected/7171f5cd-a37b-499f-a06a-2a9b4a121da2-kube-api-access-jfllw\") pod \"maas-api-revoke-sub-18b65dbd-bxxqx\" (UID: \"7171f5cd-a37b-499f-a06a-2a9b4a121da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b65dbd-bxxqx"
Oct 30 21:51:12.670131 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:12.670100    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:51:12.674943 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:12.674917    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18b65dbd-bxxqx"]
Oct 30 21:51:12.728466 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:12.728430    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7171f5cd-a37b-499f-a06a-2a9b4a121da2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18b65dbd-bxxqx\" (UID: \"7171f5cd-a37b-499f-a06a-2a9b4a121da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b65dbd-bxxqx"
Oct 30 21:51:12.728637 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:12.728497    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7171f5cd-a37b-499f-a06a-2a9b4a121da2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18b65dbd-bxxqx\" (UID: \"7171f5cd-a37b-499f-a06a-2a9b4a121da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b65dbd-bxxqx"
Oct 30 21:51:12.728637 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:12.728604    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfllw\" (UniqueName: \"kubernetes.io/projected/7171f5cd-a37b-499f-a06a-2a9b4a121da2-kube-api-access-jfllw\") pod \"maas-api-revoke-sub-18b65dbd-bxxqx\" (UID: \"7171f5cd-a37b-499f-a06a-2a9b4a121da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b65dbd-bxxqx"
Oct 30 21:51:12.729149 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:12.729124    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7171f5cd-a37b-499f-a06a-2a9b4a121da2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18b65dbd-bxxqx\" (UID: \"7171f5cd-a37b-499f-a06a-2a9b4a121da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b65dbd-bxxqx"
Oct 30 21:51:12.731200 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:12.731177    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7171f5cd-a37b-499f-a06a-2a9b4a121da2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18b65dbd-bxxqx\" (UID: \"7171f5cd-a37b-499f-a06a-2a9b4a121da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b65dbd-bxxqx"
Oct 30 21:51:12.786684 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:12.786656    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfllw\" (UniqueName: \"kubernetes.io/projected/7171f5cd-a37b-499f-a06a-2a9b4a121da2-kube-api-access-jfllw\") pod \"maas-api-revoke-sub-18b65dbd-bxxqx\" (UID: \"7171f5cd-a37b-499f-a06a-2a9b4a121da2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b65dbd-bxxqx"
Oct 30 21:51:12.880406 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:12.880315    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b65dbd-bxxqx"
Oct 30 21:51:13.255106 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:13.254937    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18b65dbd-bxxqx"]
Oct 30 21:51:13.257655 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:51:13.257631    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7171f5cd_a37b_499f_a06a_2a9b4a121da2.slice/crio-852cc4c07d8b905b71a7da5337289de9789f566a565e933ee3b8c6c36aa0f101 WatchSource:0}: Error finding container 852cc4c07d8b905b71a7da5337289de9789f566a565e933ee3b8c6c36aa0f101: Status 404 returned error can't find the container with id 852cc4c07d8b905b71a7da5337289de9789f566a565e933ee3b8c6c36aa0f101
Oct 30 21:51:14.236672 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:14.236639    5087 generic.go:356] "Generic (PLEG): container finished" podID="7171f5cd-a37b-499f-a06a-2a9b4a121da2" containerID="cd74c05cfcb78e15094aaac7f35f5b1b648f54f4d52ee1e6e2de882b86c23c85" exitCode=0
Oct 30 21:51:14.237104 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:14.236731    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b65dbd-bxxqx" event={"ID":"7171f5cd-a37b-499f-a06a-2a9b4a121da2","Type":"ContainerDied","Data":"cd74c05cfcb78e15094aaac7f35f5b1b648f54f4d52ee1e6e2de882b86c23c85"}
Oct 30 21:51:14.237104 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:14.236797    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b65dbd-bxxqx" event={"ID":"7171f5cd-a37b-499f-a06a-2a9b4a121da2","Type":"ContainerStarted","Data":"852cc4c07d8b905b71a7da5337289de9789f566a565e933ee3b8c6c36aa0f101"}
Oct 30 21:51:15.248935 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:15.248907    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e360879d-txmm9"]
Oct 30 21:51:15.253666 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:15.253646    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360879d-txmm9"
Oct 30 21:51:15.314059 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:15.314027    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e360879d-txmm9"]
Oct 30 21:51:15.358266 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:15.358232    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c2hx\" (UniqueName: \"kubernetes.io/projected/d6d4959a-1688-4683-be56-a7e208105df2-kube-api-access-8c2hx\") pod \"maas-api-revoke-sub-e360879d-txmm9\" (UID: \"d6d4959a-1688-4683-be56-a7e208105df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360879d-txmm9"
Oct 30 21:51:15.358490 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:15.358304    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6d4959a-1688-4683-be56-a7e208105df2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e360879d-txmm9\" (UID: \"d6d4959a-1688-4683-be56-a7e208105df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360879d-txmm9"
Oct 30 21:51:15.358490 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:15.358473    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6d4959a-1688-4683-be56-a7e208105df2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e360879d-txmm9\" (UID: \"d6d4959a-1688-4683-be56-a7e208105df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360879d-txmm9"
Oct 30 21:51:15.388066 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:15.388037    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b65dbd-bxxqx"
Oct 30 21:51:15.459707 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:15.459672    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7171f5cd-a37b-499f-a06a-2a9b4a121da2-kube-api-access-jfllw\" (UniqueName: \"kubernetes.io/projected/7171f5cd-a37b-499f-a06a-2a9b4a121da2-kube-api-access-jfllw\") pod \"7171f5cd-a37b-499f-a06a-2a9b4a121da2\" (UID: \"7171f5cd-a37b-499f-a06a-2a9b4a121da2\") "
Oct 30 21:51:15.459876 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:15.459737    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7171f5cd-a37b-499f-a06a-2a9b4a121da2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7171f5cd-a37b-499f-a06a-2a9b4a121da2-maas-api-xks-ca\") pod \"7171f5cd-a37b-499f-a06a-2a9b4a121da2\" (UID: \"7171f5cd-a37b-499f-a06a-2a9b4a121da2\") "
Oct 30 21:51:15.459876 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:15.459784    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7171f5cd-a37b-499f-a06a-2a9b4a121da2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7171f5cd-a37b-499f-a06a-2a9b4a121da2-maas-api-openshift-ca\") pod \"7171f5cd-a37b-499f-a06a-2a9b4a121da2\" (UID: \"7171f5cd-a37b-499f-a06a-2a9b4a121da2\") "
Oct 30 21:51:15.460092 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:15.460072    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c2hx\" (UniqueName: \"kubernetes.io/projected/d6d4959a-1688-4683-be56-a7e208105df2-kube-api-access-8c2hx\") pod \"maas-api-revoke-sub-e360879d-txmm9\" (UID: \"d6d4959a-1688-4683-be56-a7e208105df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360879d-txmm9"
Oct 30 21:51:15.460170 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:15.460147    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6d4959a-1688-4683-be56-a7e208105df2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e360879d-txmm9\" (UID: \"d6d4959a-1688-4683-be56-a7e208105df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360879d-txmm9"
Oct 30 21:51:15.460317 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:15.460180    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7171f5cd-a37b-499f-a06a-2a9b4a121da2-maas-api-openshift-ca" pod "7171f5cd-a37b-499f-a06a-2a9b4a121da2" (UID: "7171f5cd-a37b-499f-a06a-2a9b4a121da2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:51:15.460317 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:15.460193    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6d4959a-1688-4683-be56-a7e208105df2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e360879d-txmm9\" (UID: \"d6d4959a-1688-4683-be56-a7e208105df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360879d-txmm9"
Oct 30 21:51:15.460317 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:15.460245    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7171f5cd-a37b-499f-a06a-2a9b4a121da2-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:15.460949 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:15.460876    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6d4959a-1688-4683-be56-a7e208105df2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e360879d-txmm9\" (UID: \"d6d4959a-1688-4683-be56-a7e208105df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360879d-txmm9"
Oct 30 21:51:15.462590 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:15.462570    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6d4959a-1688-4683-be56-a7e208105df2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e360879d-txmm9\" (UID: \"d6d4959a-1688-4683-be56-a7e208105df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360879d-txmm9"
Oct 30 21:51:15.462678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:15.462572    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7171f5cd-a37b-499f-a06a-2a9b4a121da2-maas-api-xks-ca" pod "7171f5cd-a37b-499f-a06a-2a9b4a121da2" (UID: "7171f5cd-a37b-499f-a06a-2a9b4a121da2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:51:15.462678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:15.462574    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7171f5cd-a37b-499f-a06a-2a9b4a121da2-kube-api-access-jfllw" pod "7171f5cd-a37b-499f-a06a-2a9b4a121da2" (UID: "7171f5cd-a37b-499f-a06a-2a9b4a121da2"). InnerVolumeSpecName "kube-api-access-jfllw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:51:15.517244 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:15.517170    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c2hx\" (UniqueName: \"kubernetes.io/projected/d6d4959a-1688-4683-be56-a7e208105df2-kube-api-access-8c2hx\") pod \"maas-api-revoke-sub-e360879d-txmm9\" (UID: \"d6d4959a-1688-4683-be56-a7e208105df2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360879d-txmm9"
Oct 30 21:51:15.561180 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:15.561144    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfllw\" (UniqueName: \"kubernetes.io/projected/7171f5cd-a37b-499f-a06a-2a9b4a121da2-kube-api-access-jfllw\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:15.561180 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:15.561174    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7171f5cd-a37b-499f-a06a-2a9b4a121da2-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:15.567901 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:15.567874    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360879d-txmm9"
Oct 30 21:51:15.942959 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:15.942937    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e360879d-txmm9"]
Oct 30 21:51:15.944772 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:51:15.944738    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d4959a_1688_4683_be56_a7e208105df2.slice/crio-5473ed09bb95b64990f22b8bc73807a3259098687a40975504d8a29a6d9c6493 WatchSource:0}: Error finding container 5473ed09bb95b64990f22b8bc73807a3259098687a40975504d8a29a6d9c6493: Status 404 returned error can't find the container with id 5473ed09bb95b64990f22b8bc73807a3259098687a40975504d8a29a6d9c6493
Oct 30 21:51:16.253432 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:16.253398    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360879d-txmm9" event={"ID":"d6d4959a-1688-4683-be56-a7e208105df2","Type":"ContainerStarted","Data":"5473ed09bb95b64990f22b8bc73807a3259098687a40975504d8a29a6d9c6493"}
Oct 30 21:51:16.255074 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:16.255052    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b65dbd-bxxqx"
Oct 30 21:51:16.255074 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:16.255062    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18b65dbd-bxxqx" event={"ID":"7171f5cd-a37b-499f-a06a-2a9b4a121da2","Type":"ContainerDied","Data":"852cc4c07d8b905b71a7da5337289de9789f566a565e933ee3b8c6c36aa0f101"}
Oct 30 21:51:16.255205 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:16.255123    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="852cc4c07d8b905b71a7da5337289de9789f566a565e933ee3b8c6c36aa0f101"
Oct 30 21:51:17.171234 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:51:17.171151    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:51:17.273700 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:17.273660    5087 generic.go:356] "Generic (PLEG): container finished" podID="d6d4959a-1688-4683-be56-a7e208105df2" containerID="2cfcfe109e5d99f838646ede81ed18906c9f868dff6e2f020d9f7dcec651ddcf" exitCode=0
Oct 30 21:51:17.274095 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:17.273749    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360879d-txmm9" event={"ID":"d6d4959a-1688-4683-be56-a7e208105df2","Type":"ContainerDied","Data":"2cfcfe109e5d99f838646ede81ed18906c9f868dff6e2f020d9f7dcec651ddcf"}
Oct 30 21:51:18.425532 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:18.425505    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360879d-txmm9"
Oct 30 21:51:18.493587 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:18.493557    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6d4959a-1688-4683-be56-a7e208105df2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6d4959a-1688-4683-be56-a7e208105df2-maas-api-openshift-ca\") pod \"d6d4959a-1688-4683-be56-a7e208105df2\" (UID: \"d6d4959a-1688-4683-be56-a7e208105df2\") "
Oct 30 21:51:18.493832 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:18.493662    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d4959a-1688-4683-be56-a7e208105df2-kube-api-access-8c2hx\" (UniqueName: \"kubernetes.io/projected/d6d4959a-1688-4683-be56-a7e208105df2-kube-api-access-8c2hx\") pod \"d6d4959a-1688-4683-be56-a7e208105df2\" (UID: \"d6d4959a-1688-4683-be56-a7e208105df2\") "
Oct 30 21:51:18.493832 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:18.493682    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6d4959a-1688-4683-be56-a7e208105df2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6d4959a-1688-4683-be56-a7e208105df2-maas-api-xks-ca\") pod \"d6d4959a-1688-4683-be56-a7e208105df2\" (UID: \"d6d4959a-1688-4683-be56-a7e208105df2\") "
Oct 30 21:51:18.494003 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:18.493977    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6d4959a-1688-4683-be56-a7e208105df2-maas-api-openshift-ca" pod "d6d4959a-1688-4683-be56-a7e208105df2" (UID: "d6d4959a-1688-4683-be56-a7e208105df2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:51:18.494098 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:18.494080    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6d4959a-1688-4683-be56-a7e208105df2-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:18.496037 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:18.496009    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6d4959a-1688-4683-be56-a7e208105df2-maas-api-xks-ca" pod "d6d4959a-1688-4683-be56-a7e208105df2" (UID: "d6d4959a-1688-4683-be56-a7e208105df2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:51:18.496173 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:18.496091    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d4959a-1688-4683-be56-a7e208105df2-kube-api-access-8c2hx" pod "d6d4959a-1688-4683-be56-a7e208105df2" (UID: "d6d4959a-1688-4683-be56-a7e208105df2"). InnerVolumeSpecName "kube-api-access-8c2hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:51:18.595079 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:18.595005    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c2hx\" (UniqueName: \"kubernetes.io/projected/d6d4959a-1688-4683-be56-a7e208105df2-kube-api-access-8c2hx\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:18.595079 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:18.595032    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6d4959a-1688-4683-be56-a7e208105df2-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:19.289363 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:19.289325    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360879d-txmm9"
Oct 30 21:51:19.289601 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:19.289382    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e360879d-txmm9" event={"ID":"d6d4959a-1688-4683-be56-a7e208105df2","Type":"ContainerDied","Data":"5473ed09bb95b64990f22b8bc73807a3259098687a40975504d8a29a6d9c6493"}
Oct 30 21:51:19.289601 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:19.289428    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5473ed09bb95b64990f22b8bc73807a3259098687a40975504d8a29a6d9c6493"
Oct 30 21:51:28.006148 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.006111    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"]
Oct 30 21:51:28.006618 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.006604    5087 cpu_manager.go:414] "containerMap: removing container" podUID="7171f5cd-a37b-499f-a06a-2a9b4a121da2" containerName="revoke-keys"
Oct 30 21:51:28.006678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.006619    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7171f5cd-a37b-499f-a06a-2a9b4a121da2" containerName="revoke-keys"
Oct 30 21:51:28.006678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.006639    5087 cpu_manager.go:414] "containerMap: removing container" podUID="d6d4959a-1688-4683-be56-a7e208105df2" containerName="revoke-keys"
Oct 30 21:51:28.006678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.006645    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6d4959a-1688-4683-be56-a7e208105df2" containerName="revoke-keys"
Oct 30 21:51:28.006763 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.006746    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="7171f5cd-a37b-499f-a06a-2a9b4a121da2" containerName="revoke-keys"
Oct 30 21:51:28.006763 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.006755    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6d4959a-1688-4683-be56-a7e208105df2" containerName="revoke-keys"
Oct 30 21:51:28.011303 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.011287    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:28.063203 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.063174    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-80c366-openshift-default-dockercfg-jqtkq\""
Oct 30 21:51:28.103989 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.103951    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istiod-ca-cert\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:28.103989 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.103994    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istio-token\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:28.104189 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.104031    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-workload-socket\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:28.104189 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.104065    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz6bj\" (UniqueName: \"kubernetes.io/projected/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-kube-api-access-tz6bj\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:28.104189 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.104143    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istio-data\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:28.104189 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.104173    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istio-envoy\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:28.104315 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.104270    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-credential-socket\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:28.104385 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.104319    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istio-podinfo\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:28.104433 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.104416    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-workload-certs\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:28.116934 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.116902    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"]
Oct 30 21:51:28.205234 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.205204    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-credential-socket\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:28.205431 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.205264    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istio-podinfo\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:28.205431 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.205305    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-workload-certs\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:28.205431 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.205395    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istiod-ca-cert\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:28.205431 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.205427    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istio-token\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:28.205670 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.205463    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-workload-socket\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:28.205723 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.205663    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz6bj\" (UniqueName: \"kubernetes.io/projected/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-kube-api-access-tz6bj\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:28.205723 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.205706    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-credential-socket\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:28.205822 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.205765    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istio-data\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:28.205822 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.205775    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-workload-certs\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:28.205822 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.205794    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-workload-socket\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:28.205822 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.205805    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istio-envoy\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:28.206071 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.206050    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istio-data\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:28.206325 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.206307    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istiod-ca-cert\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:28.207991 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.207974    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istio-podinfo\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:28.207991 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.207987    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istio-envoy\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:28.263003 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.262949    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istio-token\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:28.263256 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.263234    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz6bj\" (UniqueName: \"kubernetes.io/projected/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-kube-api-access-tz6bj\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:28.324893 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.324866    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:28.754828 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.754805    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"]
Oct 30 21:51:28.757322 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:51:28.757296    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146b54d0_dc9b_4d22_9c30_fe25bd1d5194.slice/crio-19e5311fb39a65431f1ad0523aaffb1e7a4310b54367fbe1498f7e90100cdf18 WatchSource:0}: Error finding container 19e5311fb39a65431f1ad0523aaffb1e7a4310b54367fbe1498f7e90100cdf18: Status 404 returned error can't find the container with id 19e5311fb39a65431f1ad0523aaffb1e7a4310b54367fbe1498f7e90100cdf18
Oct 30 21:51:28.760426 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.760395    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:51:28.760526 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.760455    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:51:28.760526 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:28.760489    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:51:29.367300 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:29.367258    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7" event={"ID":"146b54d0-dc9b-4d22-9c30-fe25bd1d5194","Type":"ContainerStarted","Data":"92fff8afe405ce49e361b133b95ed80dd7e4ab927054e393d22924b42f9ea685"}
Oct 30 21:51:29.367857 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:29.367326    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7" event={"ID":"146b54d0-dc9b-4d22-9c30-fe25bd1d5194","Type":"ContainerStarted","Data":"19e5311fb39a65431f1ad0523aaffb1e7a4310b54367fbe1498f7e90100cdf18"}
Oct 30 21:51:29.484279 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:29.484220    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7" podStartSLOduration=2.484207816 podStartE2EDuration="2.484207816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:29.483367866 +0000 UTC m=+1083.957871835" watchObservedRunningTime="2026-10-30 21:51:29.484207816 +0000 UTC m=+1083.958711850"
Oct 30 21:51:30.325394 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:30.325336    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:30.326869 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:30.326841    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:51:30.327019 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:30.326898    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7" podUID="146b54d0-dc9b-4d22-9c30-fe25bd1d5194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 21:51:30.469155 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:30.469108    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5aa62eb-rw7kj"]
Oct 30 21:51:30.473612 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:30.473592    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aa62eb-rw7kj"
Oct 30 21:51:30.528768 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:30.528739    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5aa62eb-rw7kj"]
Oct 30 21:51:30.530609 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:30.530585    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d82ef9ca-3cfa-4141-8669-af559a5384c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5aa62eb-rw7kj\" (UID: \"d82ef9ca-3cfa-4141-8669-af559a5384c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aa62eb-rw7kj"
Oct 30 21:51:30.530734 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:30.530622    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2bgl\" (UniqueName: \"kubernetes.io/projected/d82ef9ca-3cfa-4141-8669-af559a5384c0-kube-api-access-q2bgl\") pod \"maas-api-revoke-sub-d5aa62eb-rw7kj\" (UID: \"d82ef9ca-3cfa-4141-8669-af559a5384c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aa62eb-rw7kj"
Oct 30 21:51:30.530793 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:30.530746    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d82ef9ca-3cfa-4141-8669-af559a5384c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5aa62eb-rw7kj\" (UID: \"d82ef9ca-3cfa-4141-8669-af559a5384c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aa62eb-rw7kj"
Oct 30 21:51:30.573678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:30.573646    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:51:30.576203 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:30.574014    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:51:30.632415 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:30.632391    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d82ef9ca-3cfa-4141-8669-af559a5384c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5aa62eb-rw7kj\" (UID: \"d82ef9ca-3cfa-4141-8669-af559a5384c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aa62eb-rw7kj"
Oct 30 21:51:30.632609 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:30.632595    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2bgl\" (UniqueName: \"kubernetes.io/projected/d82ef9ca-3cfa-4141-8669-af559a5384c0-kube-api-access-q2bgl\") pod \"maas-api-revoke-sub-d5aa62eb-rw7kj\" (UID: \"d82ef9ca-3cfa-4141-8669-af559a5384c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aa62eb-rw7kj"
Oct 30 21:51:30.633099 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:30.633078    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d82ef9ca-3cfa-4141-8669-af559a5384c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5aa62eb-rw7kj\" (UID: \"d82ef9ca-3cfa-4141-8669-af559a5384c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aa62eb-rw7kj"
Oct 30 21:51:30.633794 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:30.633677    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d82ef9ca-3cfa-4141-8669-af559a5384c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5aa62eb-rw7kj\" (UID: \"d82ef9ca-3cfa-4141-8669-af559a5384c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aa62eb-rw7kj"
Oct 30 21:51:30.634921 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:30.634900    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d82ef9ca-3cfa-4141-8669-af559a5384c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5aa62eb-rw7kj\" (UID: \"d82ef9ca-3cfa-4141-8669-af559a5384c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aa62eb-rw7kj"
Oct 30 21:51:30.691925 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:30.691882    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2bgl\" (UniqueName: \"kubernetes.io/projected/d82ef9ca-3cfa-4141-8669-af559a5384c0-kube-api-access-q2bgl\") pod \"maas-api-revoke-sub-d5aa62eb-rw7kj\" (UID: \"d82ef9ca-3cfa-4141-8669-af559a5384c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aa62eb-rw7kj"
Oct 30 21:51:30.788838 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:30.788813    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aa62eb-rw7kj"
Oct 30 21:51:31.196502 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:31.196476    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5aa62eb-rw7kj"]
Oct 30 21:51:31.197639 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:51:31.197608    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd82ef9ca_3cfa_4141_8669_af559a5384c0.slice/crio-2c5df9170a6c11c824c55e9411d8209b78d3cc6dcae72632f90c70f9599eb6fd WatchSource:0}: Error finding container 2c5df9170a6c11c824c55e9411d8209b78d3cc6dcae72632f90c70f9599eb6fd: Status 404 returned error can't find the container with id 2c5df9170a6c11c824c55e9411d8209b78d3cc6dcae72632f90c70f9599eb6fd
Oct 30 21:51:31.326013 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:31.325973    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:51:31.326175 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:31.326032    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7" podUID="146b54d0-dc9b-4d22-9c30-fe25bd1d5194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 21:51:31.383326 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:31.383288    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aa62eb-rw7kj" event={"ID":"d82ef9ca-3cfa-4141-8669-af559a5384c0","Type":"ContainerStarted","Data":"2c5df9170a6c11c824c55e9411d8209b78d3cc6dcae72632f90c70f9599eb6fd"}
Oct 30 21:51:32.325153 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:32.325117    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:51:32.325541 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:32.325175    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7" podUID="146b54d0-dc9b-4d22-9c30-fe25bd1d5194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 21:51:32.390459 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:32.390421    5087 generic.go:356] "Generic (PLEG): container finished" podID="d82ef9ca-3cfa-4141-8669-af559a5384c0" containerID="1bb762b202bd7d179192284710ab1142fea012d136cbf7dfcc3bbbc2fd1e86ce" exitCode=0
Oct 30 21:51:32.390642 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:32.390503    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aa62eb-rw7kj" event={"ID":"d82ef9ca-3cfa-4141-8669-af559a5384c0","Type":"ContainerDied","Data":"1bb762b202bd7d179192284710ab1142fea012d136cbf7dfcc3bbbc2fd1e86ce"}
Oct 30 21:51:33.325529 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:33.325496    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:51:33.325905 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:33.325549    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7" podUID="146b54d0-dc9b-4d22-9c30-fe25bd1d5194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 21:51:33.610623 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:33.610576    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aa62eb-rw7kj"
Oct 30 21:51:33.791908 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:33.791870    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d82ef9ca-3cfa-4141-8669-af559a5384c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d82ef9ca-3cfa-4141-8669-af559a5384c0-maas-api-xks-ca\") pod \"d82ef9ca-3cfa-4141-8669-af559a5384c0\" (UID: \"d82ef9ca-3cfa-4141-8669-af559a5384c0\") "
Oct 30 21:51:33.792106 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:33.791933    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d82ef9ca-3cfa-4141-8669-af559a5384c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d82ef9ca-3cfa-4141-8669-af559a5384c0-maas-api-openshift-ca\") pod \"d82ef9ca-3cfa-4141-8669-af559a5384c0\" (UID: \"d82ef9ca-3cfa-4141-8669-af559a5384c0\") "
Oct 30 21:51:33.792106 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:33.791967    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d82ef9ca-3cfa-4141-8669-af559a5384c0-kube-api-access-q2bgl\" (UniqueName: \"kubernetes.io/projected/d82ef9ca-3cfa-4141-8669-af559a5384c0-kube-api-access-q2bgl\") pod \"d82ef9ca-3cfa-4141-8669-af559a5384c0\" (UID: \"d82ef9ca-3cfa-4141-8669-af559a5384c0\") "
Oct 30 21:51:33.792771 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:33.792720    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d82ef9ca-3cfa-4141-8669-af559a5384c0-maas-api-openshift-ca" pod "d82ef9ca-3cfa-4141-8669-af559a5384c0" (UID: "d82ef9ca-3cfa-4141-8669-af559a5384c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:51:33.794674 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:33.794649    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d82ef9ca-3cfa-4141-8669-af559a5384c0-maas-api-xks-ca" pod "d82ef9ca-3cfa-4141-8669-af559a5384c0" (UID: "d82ef9ca-3cfa-4141-8669-af559a5384c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:51:33.796388 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:33.796360    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d82ef9ca-3cfa-4141-8669-af559a5384c0-kube-api-access-q2bgl" pod "d82ef9ca-3cfa-4141-8669-af559a5384c0" (UID: "d82ef9ca-3cfa-4141-8669-af559a5384c0"). InnerVolumeSpecName "kube-api-access-q2bgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:51:33.893028 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:33.892949    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d82ef9ca-3cfa-4141-8669-af559a5384c0-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:33.893028 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:33.892983    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d82ef9ca-3cfa-4141-8669-af559a5384c0-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:33.893028 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:33.893000    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2bgl\" (UniqueName: \"kubernetes.io/projected/d82ef9ca-3cfa-4141-8669-af559a5384c0-kube-api-access-q2bgl\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:51:34.325929 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:34.325886    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 30 21:51:34.326328 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:34.325956    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7" podUID="146b54d0-dc9b-4d22-9c30-fe25bd1d5194" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 30 21:51:34.406391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:34.406335    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aa62eb-rw7kj" event={"ID":"d82ef9ca-3cfa-4141-8669-af559a5384c0","Type":"ContainerDied","Data":"2c5df9170a6c11c824c55e9411d8209b78d3cc6dcae72632f90c70f9599eb6fd"}
Oct 30 21:51:34.406568 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:34.406409    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5aa62eb-rw7kj"
Oct 30 21:51:34.406568 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:34.406411    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c5df9170a6c11c824c55e9411d8209b78d3cc6dcae72632f90c70f9599eb6fd"
Oct 30 21:51:35.540438 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:35.540407    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:35.541587 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:35.541558    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:35.542955 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:35.542939    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 21:51:38.491041 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:38.491005    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-jlrnh-qhkzt"]
Oct 30 21:51:38.491594 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:38.491578    5087 cpu_manager.go:414] "containerMap: removing container" podUID="d82ef9ca-3cfa-4141-8669-af559a5384c0" containerName="revoke-keys"
Oct 30 21:51:38.491654 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:38.491595    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d82ef9ca-3cfa-4141-8669-af559a5384c0" containerName="revoke-keys"
Oct 30 21:51:38.491690 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:38.491685    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="d82ef9ca-3cfa-4141-8669-af559a5384c0" containerName="revoke-keys"
Oct 30 21:51:38.495252 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:38.495238    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jlrnh-qhkzt"
Oct 30 21:51:38.543718 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:38.543689    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49-archives-path\") pod \"periodic-gathering-jlrnh-qhkzt\" (UID: \"f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49\") " pod="openshift-insights/periodic-gathering-jlrnh-qhkzt"
Oct 30 21:51:38.543856 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:38.543738    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2h9p\" (UniqueName: \"kubernetes.io/projected/f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49-kube-api-access-m2h9p\") pod \"periodic-gathering-jlrnh-qhkzt\" (UID: \"f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49\") " pod="openshift-insights/periodic-gathering-jlrnh-qhkzt"
Oct 30 21:51:38.543913 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:38.543873    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49-service-ca-bundle\") pod \"periodic-gathering-jlrnh-qhkzt\" (UID: \"f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49\") " pod="openshift-insights/periodic-gathering-jlrnh-qhkzt"
Oct 30 21:51:38.543913 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:38.543898    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49-trusted-ca-bundle\") pod \"periodic-gathering-jlrnh-qhkzt\" (UID: \"f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49\") " pod="openshift-insights/periodic-gathering-jlrnh-qhkzt"
Oct 30 21:51:38.553635 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:38.553606    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-jlrnh-qhkzt"]
Oct 30 21:51:38.644797 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:38.644764    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49-service-ca-bundle\") pod \"periodic-gathering-jlrnh-qhkzt\" (UID: \"f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49\") " pod="openshift-insights/periodic-gathering-jlrnh-qhkzt"
Oct 30 21:51:38.644797 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:38.644804    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49-trusted-ca-bundle\") pod \"periodic-gathering-jlrnh-qhkzt\" (UID: \"f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49\") " pod="openshift-insights/periodic-gathering-jlrnh-qhkzt"
Oct 30 21:51:38.645003 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:38.644858    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49-archives-path\") pod \"periodic-gathering-jlrnh-qhkzt\" (UID: \"f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49\") " pod="openshift-insights/periodic-gathering-jlrnh-qhkzt"
Oct 30 21:51:38.645003 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:38.644884    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2h9p\" (UniqueName: \"kubernetes.io/projected/f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49-kube-api-access-m2h9p\") pod \"periodic-gathering-jlrnh-qhkzt\" (UID: \"f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49\") " pod="openshift-insights/periodic-gathering-jlrnh-qhkzt"
Oct 30 21:51:38.645332 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:38.645312    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49-archives-path\") pod \"periodic-gathering-jlrnh-qhkzt\" (UID: \"f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49\") " pod="openshift-insights/periodic-gathering-jlrnh-qhkzt"
Oct 30 21:51:38.645658 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:38.645634    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49-service-ca-bundle\") pod \"periodic-gathering-jlrnh-qhkzt\" (UID: \"f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49\") " pod="openshift-insights/periodic-gathering-jlrnh-qhkzt"
Oct 30 21:51:38.645702 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:38.645689    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49-trusted-ca-bundle\") pod \"periodic-gathering-jlrnh-qhkzt\" (UID: \"f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49\") " pod="openshift-insights/periodic-gathering-jlrnh-qhkzt"
Oct 30 21:51:38.703002 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:38.702977    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2h9p\" (UniqueName: \"kubernetes.io/projected/f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49-kube-api-access-m2h9p\") pod \"periodic-gathering-jlrnh-qhkzt\" (UID: \"f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49\") " pod="openshift-insights/periodic-gathering-jlrnh-qhkzt"
Oct 30 21:51:38.808039 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:38.807957    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jlrnh-qhkzt"
Oct 30 21:51:39.208768 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:39.208738    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-jlrnh-qhkzt"]
Oct 30 21:51:39.211202 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:51:39.211176    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf44dd57f_7e32_4ccd_b2d7_aeb7bb923c49.slice/crio-b12ddd34de77f0dd068a47f65cd48aa381cbe11a4ccf7af9009cbdb8d8b03662 WatchSource:0}: Error finding container b12ddd34de77f0dd068a47f65cd48aa381cbe11a4ccf7af9009cbdb8d8b03662: Status 404 returned error can't find the container with id b12ddd34de77f0dd068a47f65cd48aa381cbe11a4ccf7af9009cbdb8d8b03662
Oct 30 21:51:39.442206 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:39.442163    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jlrnh-qhkzt" event={"ID":"f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49","Type":"ContainerStarted","Data":"b9516405757309c69ef0c7f3e9408631317a07932c3b11c0e49661dbe847c67e"}
Oct 30 21:51:39.442442 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:39.442230    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jlrnh-qhkzt" event={"ID":"f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49","Type":"ContainerStarted","Data":"b12ddd34de77f0dd068a47f65cd48aa381cbe11a4ccf7af9009cbdb8d8b03662"}
Oct 30 21:51:39.558647 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:39.558246    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-jlrnh-qhkzt" podStartSLOduration=1.558228309 podStartE2EDuration="1.558228309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:39.55794271 +0000 UTC m=+1094.032446690" watchObservedRunningTime="2026-10-30 21:51:39.558228309 +0000 UTC m=+1094.032732289"
Oct 30 21:51:42.221741 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.221706    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"]
Oct 30 21:51:42.226599 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.226574    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:42.287860 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.287827    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istiod-ca-cert\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:42.288007 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.287868    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-workload-socket\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:42.288007 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.287885    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-credential-socket\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:42.288007 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.287909    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istio-data\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:42.288155 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.288010    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istio-envoy\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:42.288155 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.288077    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istio-token\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:42.288267 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.288215    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjnw4\" (UniqueName: \"kubernetes.io/projected/dafd13b7-c191-4d83-b94c-371bf53a9fe5-kube-api-access-bjnw4\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:42.288267 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.288260    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istio-podinfo\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:42.288393 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.288373    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-workload-certs\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:42.315449 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.315423    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"]
Oct 30 21:51:42.389380 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.389336    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istio-token\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:42.389522 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.389439    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjnw4\" (UniqueName: \"kubernetes.io/projected/dafd13b7-c191-4d83-b94c-371bf53a9fe5-kube-api-access-bjnw4\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:42.389522 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.389473    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istio-podinfo\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:42.389612 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.389531    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-workload-certs\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:42.389612 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.389562    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istiod-ca-cert\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:42.389612 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.389603    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-workload-socket\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:42.389760 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.389630    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-credential-socket\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:42.389760 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.389655    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istio-data\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:42.389760 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.389704    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istio-envoy\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:42.390013 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.389993    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-workload-socket\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:42.390156 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.390117    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-workload-certs\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:42.390276 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.390155    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istio-data\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:42.390276 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.390215    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-credential-socket\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:42.390476 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.390454    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istiod-ca-cert\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:42.392007 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.391985    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istio-envoy\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:42.392288 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.392268    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istio-podinfo\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:42.446472 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.446452    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjnw4\" (UniqueName: \"kubernetes.io/projected/dafd13b7-c191-4d83-b94c-371bf53a9fe5-kube-api-access-bjnw4\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:42.446600 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.446583    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istio-token\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:42.540304 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.540277    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:42.976097 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.976070    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"]
Oct 30 21:51:42.976675 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:51:42.976647    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddafd13b7_c191_4d83_b94c_371bf53a9fe5.slice/crio-777860780e80cc85966481c69fd5d53755258cb3e476a8fb334230067a9b7bac WatchSource:0}: Error finding container 777860780e80cc85966481c69fd5d53755258cb3e476a8fb334230067a9b7bac: Status 404 returned error can't find the container with id 777860780e80cc85966481c69fd5d53755258cb3e476a8fb334230067a9b7bac
Oct 30 21:51:42.979775 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.979743    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:51:42.979861 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.979809    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:51:42.979861 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:42.979844    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:51:43.453780 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.453740    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"]
Oct 30 21:51:43.464644 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.464613    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:43.476572 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.476539    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc" event={"ID":"dafd13b7-c191-4d83-b94c-371bf53a9fe5","Type":"ContainerStarted","Data":"7c16092a1b86edc469898c83c4d35c267aa765c3ceb5b2a3c0171b29ef51687a"}
Oct 30 21:51:43.476744 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.476600    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc" event={"ID":"dafd13b7-c191-4d83-b94c-371bf53a9fe5","Type":"ContainerStarted","Data":"777860780e80cc85966481c69fd5d53755258cb3e476a8fb334230067a9b7bac"}
Oct 30 21:51:43.501893 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.501863    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-workload-certs\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:43.502032 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.501900    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ef40b8-03ce-449b-ab82-da0b7af9e155-istiod-ca-cert\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:43.502032 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.502007    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-istio-envoy\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:43.502135 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.502045    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-credential-socket\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:43.502135 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.502105    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-istio-data\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:43.502213 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.502149    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rhz7\" (UniqueName: \"kubernetes.io/projected/66ef40b8-03ce-449b-ab82-da0b7af9e155-kube-api-access-5rhz7\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:43.502213 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.502195    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ef40b8-03ce-449b-ab82-da0b7af9e155-istio-token\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:43.502323 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.502250    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-workload-socket\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:43.502412 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.502332    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ef40b8-03ce-449b-ab82-da0b7af9e155-istio-podinfo\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:43.518835 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.518810    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"]
Oct 30 21:51:43.522327 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.522298    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-9b51e1-openshift-default-dockercfg-b9z85\""
Oct 30 21:51:43.567386 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.566616    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:43.602987 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.602958    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ef40b8-03ce-449b-ab82-da0b7af9e155-istio-podinfo\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:43.603185 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.603005    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-workload-certs\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:43.603185 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.603042    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ef40b8-03ce-449b-ab82-da0b7af9e155-istiod-ca-cert\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:43.603185 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.603126    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-istio-envoy\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:43.603185 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.603155    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-credential-socket\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:43.603185 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.603177    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-istio-data\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:43.603573 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.603225    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rhz7\" (UniqueName: \"kubernetes.io/projected/66ef40b8-03ce-449b-ab82-da0b7af9e155-kube-api-access-5rhz7\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:43.603573 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.603328    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ef40b8-03ce-449b-ab82-da0b7af9e155-istio-token\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:43.603573 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.603407    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-workload-socket\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:43.603573 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.603525    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-workload-certs\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:43.603573 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.603554    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-credential-socket\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:43.603823 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.603764    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-workload-socket\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:43.603882 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.603850    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-istio-data\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:43.604138 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.604072    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ef40b8-03ce-449b-ab82-da0b7af9e155-istiod-ca-cert\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:43.605773 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.605750    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-istio-envoy\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:43.606057 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.606042    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ef40b8-03ce-449b-ab82-da0b7af9e155-istio-podinfo\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:43.664818 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.664784    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ef40b8-03ce-449b-ab82-da0b7af9e155-istio-token\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:43.664996 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.664875    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rhz7\" (UniqueName: \"kubernetes.io/projected/66ef40b8-03ce-449b-ab82-da0b7af9e155-kube-api-access-5rhz7\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:43.766084 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.766029    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc" podStartSLOduration=1.7660139209999999 podStartE2EDuration="1.766013921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:43.761964656 +0000 UTC m=+1098.236468635" watchObservedRunningTime="2026-10-30 21:51:43.766013921 +0000 UTC m=+1098.240517899"
Oct 30 21:51:43.785690 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:43.785664    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:44.268592 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:44.268559    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"]
Oct 30 21:51:44.269574 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:51:44.269530    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ef40b8_03ce_449b_ab82_da0b7af9e155.slice/crio-7e26218d04f2a3dfc8cd3269e58b2dde181e1447950d427a5863c10ba8df8690 WatchSource:0}: Error finding container 7e26218d04f2a3dfc8cd3269e58b2dde181e1447950d427a5863c10ba8df8690: Status 404 returned error can't find the container with id 7e26218d04f2a3dfc8cd3269e58b2dde181e1447950d427a5863c10ba8df8690
Oct 30 21:51:44.272066 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:44.272035    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:51:44.272148 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:44.272099    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:51:44.272148 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:44.272128    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:51:44.485017 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:44.484980    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf" event={"ID":"66ef40b8-03ce-449b-ab82-da0b7af9e155","Type":"ContainerStarted","Data":"0015000afa9c7705810d6c864ba040b04ec2ff91cbfed8ba9920266178cc9f7b"}
Oct 30 21:51:44.485522 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:44.485038    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf" event={"ID":"66ef40b8-03ce-449b-ab82-da0b7af9e155","Type":"ContainerStarted","Data":"7e26218d04f2a3dfc8cd3269e58b2dde181e1447950d427a5863c10ba8df8690"}
Oct 30 21:51:44.566417 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:44.566302    5087 patch_prober.go:29] interesting pod/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:51:44.566417 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:44.566385    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc" podUID="dafd13b7-c191-4d83-b94c-371bf53a9fe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 21:51:44.629613 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:44.629564    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf" podStartSLOduration=1.629547715 podStartE2EDuration="1.629547715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:44.62925021 +0000 UTC m=+1099.103754203" watchObservedRunningTime="2026-10-30 21:51:44.629547715 +0000 UTC m=+1099.104051694"
Oct 30 21:51:44.785874 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:44.785845    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:45.541371 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:45.540664    5087 patch_prober.go:29] interesting pod/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:51:45.541371 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:45.540718    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc" podUID="dafd13b7-c191-4d83-b94c-371bf53a9fe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 21:51:45.799429 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:45.798816    5087 patch_prober.go:29] interesting pod/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 21:51:45.799429 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:45.798882    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf" podUID="66ef40b8-03ce-449b-ab82-da0b7af9e155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 21:51:46.555373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:46.542779    5087 patch_prober.go:29] interesting pod/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:51:46.555373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:46.542854    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc" podUID="dafd13b7-c191-4d83-b94c-371bf53a9fe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 21:51:46.787371 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:46.786730    5087 patch_prober.go:29] interesting pod/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 21:51:46.787371 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:46.786804    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf" podUID="66ef40b8-03ce-449b-ab82-da0b7af9e155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 21:51:47.217084 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:51:47.216636    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:51:47.550380 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:47.549690    5087 patch_prober.go:29] interesting pod/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:51:47.550380 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:47.549752    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc" podUID="dafd13b7-c191-4d83-b94c-371bf53a9fe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 21:51:47.709719 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:47.709687    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"]
Oct 30 21:51:47.713708 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:47.713692    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:51:47.790372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:47.789759    5087 patch_prober.go:29] interesting pod/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 21:51:47.790372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:47.789817    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf" podUID="66ef40b8-03ce-449b-ab82-da0b7af9e155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 21:51:47.796711 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:47.796677    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"]
Oct 30 21:51:47.797313 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:47.797289    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/777cc409-27c5-48c2-9f31-8f64bc4744dc-dshm\") pod \"distinct-w0-dda9f5-kserve-b9946d498-tfpgt\" (UID: \"777cc409-27c5-48c2-9f31-8f64bc4744dc\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:51:47.797534 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:47.797507    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/777cc409-27c5-48c2-9f31-8f64bc4744dc-home\") pod \"distinct-w0-dda9f5-kserve-b9946d498-tfpgt\" (UID: \"777cc409-27c5-48c2-9f31-8f64bc4744dc\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:51:47.797680 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:47.797667    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/777cc409-27c5-48c2-9f31-8f64bc4744dc-tmp-dir\") pod \"distinct-w0-dda9f5-kserve-b9946d498-tfpgt\" (UID: \"777cc409-27c5-48c2-9f31-8f64bc4744dc\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:51:47.797896 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:47.797880    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t89cb\" (UniqueName: \"kubernetes.io/projected/777cc409-27c5-48c2-9f31-8f64bc4744dc-kube-api-access-t89cb\") pod \"distinct-w0-dda9f5-kserve-b9946d498-tfpgt\" (UID: \"777cc409-27c5-48c2-9f31-8f64bc4744dc\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:51:47.798077 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:47.798065    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/777cc409-27c5-48c2-9f31-8f64bc4744dc-model-cache\") pod \"distinct-w0-dda9f5-kserve-b9946d498-tfpgt\" (UID: \"777cc409-27c5-48c2-9f31-8f64bc4744dc\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:51:47.798186 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:47.798174    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/777cc409-27c5-48c2-9f31-8f64bc4744dc-tls-certs\") pod \"distinct-w0-dda9f5-kserve-b9946d498-tfpgt\" (UID: \"777cc409-27c5-48c2-9f31-8f64bc4744dc\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:51:47.819533 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:47.819466    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-dda9f5\"/\"openshift-service-ca.crt\""
Oct 30 21:51:47.819533 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:47.819466    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-dda9f5\"/\"default-dockercfg-47mqx\""
Oct 30 21:51:47.819533 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:47.819528    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-dda9f5\"/\"distinct-w0-dda9f5-kserve-self-signed-certs\""
Oct 30 21:51:47.819719 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:47.819602    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-dda9f5\"/\"kube-root-ca.crt\""
Oct 30 21:51:47.921371 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:47.918148    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/777cc409-27c5-48c2-9f31-8f64bc4744dc-home\") pod \"distinct-w0-dda9f5-kserve-b9946d498-tfpgt\" (UID: \"777cc409-27c5-48c2-9f31-8f64bc4744dc\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:51:47.921371 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:47.918214    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/777cc409-27c5-48c2-9f31-8f64bc4744dc-tmp-dir\") pod \"distinct-w0-dda9f5-kserve-b9946d498-tfpgt\" (UID: \"777cc409-27c5-48c2-9f31-8f64bc4744dc\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:51:47.921371 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:47.918318    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t89cb\" (UniqueName: \"kubernetes.io/projected/777cc409-27c5-48c2-9f31-8f64bc4744dc-kube-api-access-t89cb\") pod \"distinct-w0-dda9f5-kserve-b9946d498-tfpgt\" (UID: \"777cc409-27c5-48c2-9f31-8f64bc4744dc\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:51:47.923110 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:47.923053    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/777cc409-27c5-48c2-9f31-8f64bc4744dc-home\") pod \"distinct-w0-dda9f5-kserve-b9946d498-tfpgt\" (UID: \"777cc409-27c5-48c2-9f31-8f64bc4744dc\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:51:47.924388 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:47.923569    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/777cc409-27c5-48c2-9f31-8f64bc4744dc-tmp-dir\") pod \"distinct-w0-dda9f5-kserve-b9946d498-tfpgt\" (UID: \"777cc409-27c5-48c2-9f31-8f64bc4744dc\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:51:47.924685 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:47.924668    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/777cc409-27c5-48c2-9f31-8f64bc4744dc-model-cache\") pod \"distinct-w0-dda9f5-kserve-b9946d498-tfpgt\" (UID: \"777cc409-27c5-48c2-9f31-8f64bc4744dc\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:51:47.924833 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:47.924818    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/777cc409-27c5-48c2-9f31-8f64bc4744dc-tls-certs\") pod \"distinct-w0-dda9f5-kserve-b9946d498-tfpgt\" (UID: \"777cc409-27c5-48c2-9f31-8f64bc4744dc\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:51:47.925406 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:47.925386    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/777cc409-27c5-48c2-9f31-8f64bc4744dc-dshm\") pod \"distinct-w0-dda9f5-kserve-b9946d498-tfpgt\" (UID: \"777cc409-27c5-48c2-9f31-8f64bc4744dc\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:51:47.925779 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:47.925124    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/777cc409-27c5-48c2-9f31-8f64bc4744dc-model-cache\") pod \"distinct-w0-dda9f5-kserve-b9946d498-tfpgt\" (UID: \"777cc409-27c5-48c2-9f31-8f64bc4744dc\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:51:47.930173 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:47.930129    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/777cc409-27c5-48c2-9f31-8f64bc4744dc-tls-certs\") pod \"distinct-w0-dda9f5-kserve-b9946d498-tfpgt\" (UID: \"777cc409-27c5-48c2-9f31-8f64bc4744dc\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:51:47.932442 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:47.932422    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/777cc409-27c5-48c2-9f31-8f64bc4744dc-dshm\") pod \"distinct-w0-dda9f5-kserve-b9946d498-tfpgt\" (UID: \"777cc409-27c5-48c2-9f31-8f64bc4744dc\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:51:47.984375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:47.984270    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t89cb\" (UniqueName: \"kubernetes.io/projected/777cc409-27c5-48c2-9f31-8f64bc4744dc-kube-api-access-t89cb\") pod \"distinct-w0-dda9f5-kserve-b9946d498-tfpgt\" (UID: \"777cc409-27c5-48c2-9f31-8f64bc4744dc\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:51:48.027673 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:48.027634    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:51:48.558374 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:48.553288    5087 patch_prober.go:29] interesting pod/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:51:48.558374 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:48.553369    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc" podUID="dafd13b7-c191-4d83-b94c-371bf53a9fe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 21:51:48.720718 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:48.720685    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"]
Oct 30 21:51:48.722066 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:51:48.722044    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod777cc409_27c5_48c2_9f31_8f64bc4744dc.slice/crio-882eb7d446567f93857b33e9c49177d4d45a171efbd57bdb4f4004484939719f WatchSource:0}: Error finding container 882eb7d446567f93857b33e9c49177d4d45a171efbd57bdb4f4004484939719f: Status 404 returned error can't find the container with id 882eb7d446567f93857b33e9c49177d4d45a171efbd57bdb4f4004484939719f
Oct 30 21:51:48.795378 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:48.794247    5087 patch_prober.go:29] interesting pod/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 21:51:48.795378 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:48.794317    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf" podUID="66ef40b8-03ce-449b-ab82-da0b7af9e155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 21:51:49.535313 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:49.535272    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" event={"ID":"777cc409-27c5-48c2-9f31-8f64bc4744dc","Type":"ContainerStarted","Data":"40f8dbd7ac6ec213cbf496c4c8949e4952f537c82625d058b7eac43d56bccc7f"}
Oct 30 21:51:49.535522 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:49.535337    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" event={"ID":"777cc409-27c5-48c2-9f31-8f64bc4744dc","Type":"ContainerStarted","Data":"882eb7d446567f93857b33e9c49177d4d45a171efbd57bdb4f4004484939719f"}
Oct 30 21:51:49.690437 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:49.690371    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podStartSLOduration=2.690340997 podStartE2EDuration="2.690340997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:49.668176561 +0000 UTC m=+1104.142680540" watchObservedRunningTime="2026-10-30 21:51:49.690340997 +0000 UTC m=+1104.164844979"
Oct 30 21:51:49.789379 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:49.787284    5087 patch_prober.go:29] interesting pod/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 21:51:49.789379 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:49.787391    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf" podUID="66ef40b8-03ce-449b-ab82-da0b7af9e155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 21:51:49.895340 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:49.895309    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:49.896006 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:49.895988    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:49.897012 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:49.896991    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 21:51:50.028453 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:50.028425    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:51:50.786200 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:50.786157    5087 patch_prober.go:29] interesting pod/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 21:51:50.786436 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:50.786226    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf" podUID="66ef40b8-03ce-449b-ab82-da0b7af9e155" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 21:51:51.794587 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:51.794546    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:51.795728 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:51.795698    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:51.797042 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:51.797019    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 21:51:58.028684 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:58.028653    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:51:58.041465 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:58.041441    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:51:58.608492 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:51:58.608468    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:52:05.349796 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:05.349719    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"]
Oct 30 21:52:05.354962 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:05.354942    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:52:05.457277 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:05.457245    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c79cb5\"/\"distinct-w2-c79cb5-kserve-self-signed-certs\""
Oct 30 21:52:05.460802 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:05.460774    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"]
Oct 30 21:52:05.489200 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:05.489162    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07695dfe-10c8-44af-89c5-cbb19eb57ddf-model-cache\") pod \"distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg\" (UID: \"07695dfe-10c8-44af-89c5-cbb19eb57ddf\") " pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:52:05.489200 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:05.489196    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07695dfe-10c8-44af-89c5-cbb19eb57ddf-tls-certs\") pod \"distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg\" (UID: \"07695dfe-10c8-44af-89c5-cbb19eb57ddf\") " pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:52:05.489492 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:05.489217    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07695dfe-10c8-44af-89c5-cbb19eb57ddf-tmp-dir\") pod \"distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg\" (UID: \"07695dfe-10c8-44af-89c5-cbb19eb57ddf\") " pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:52:05.489492 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:05.489322    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07695dfe-10c8-44af-89c5-cbb19eb57ddf-dshm\") pod \"distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg\" (UID: \"07695dfe-10c8-44af-89c5-cbb19eb57ddf\") " pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:52:05.489492 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:05.489428    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07695dfe-10c8-44af-89c5-cbb19eb57ddf-home\") pod \"distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg\" (UID: \"07695dfe-10c8-44af-89c5-cbb19eb57ddf\") " pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:52:05.489492 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:05.489488    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwdkd\" (UniqueName: \"kubernetes.io/projected/07695dfe-10c8-44af-89c5-cbb19eb57ddf-kube-api-access-pwdkd\") pod \"distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg\" (UID: \"07695dfe-10c8-44af-89c5-cbb19eb57ddf\") " pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:52:05.591060 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:05.591025    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07695dfe-10c8-44af-89c5-cbb19eb57ddf-model-cache\") pod \"distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg\" (UID: \"07695dfe-10c8-44af-89c5-cbb19eb57ddf\") " pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:52:05.591252 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:05.591065    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07695dfe-10c8-44af-89c5-cbb19eb57ddf-tls-certs\") pod \"distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg\" (UID: \"07695dfe-10c8-44af-89c5-cbb19eb57ddf\") " pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:52:05.591252 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:05.591088    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07695dfe-10c8-44af-89c5-cbb19eb57ddf-tmp-dir\") pod \"distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg\" (UID: \"07695dfe-10c8-44af-89c5-cbb19eb57ddf\") " pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:52:05.591252 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:05.591136    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07695dfe-10c8-44af-89c5-cbb19eb57ddf-dshm\") pod \"distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg\" (UID: \"07695dfe-10c8-44af-89c5-cbb19eb57ddf\") " pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:52:05.591252 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:05.591209    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07695dfe-10c8-44af-89c5-cbb19eb57ddf-home\") pod \"distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg\" (UID: \"07695dfe-10c8-44af-89c5-cbb19eb57ddf\") " pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:52:05.591493 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:05.591279    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwdkd\" (UniqueName: \"kubernetes.io/projected/07695dfe-10c8-44af-89c5-cbb19eb57ddf-kube-api-access-pwdkd\") pod \"distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg\" (UID: \"07695dfe-10c8-44af-89c5-cbb19eb57ddf\") " pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:52:05.591585 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:05.591538    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07695dfe-10c8-44af-89c5-cbb19eb57ddf-model-cache\") pod \"distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg\" (UID: \"07695dfe-10c8-44af-89c5-cbb19eb57ddf\") " pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:52:05.591691 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:05.591592    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07695dfe-10c8-44af-89c5-cbb19eb57ddf-tmp-dir\") pod \"distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg\" (UID: \"07695dfe-10c8-44af-89c5-cbb19eb57ddf\") " pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:52:05.591691 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:05.591647    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07695dfe-10c8-44af-89c5-cbb19eb57ddf-home\") pod \"distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg\" (UID: \"07695dfe-10c8-44af-89c5-cbb19eb57ddf\") " pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:52:05.593647 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:05.593616    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07695dfe-10c8-44af-89c5-cbb19eb57ddf-dshm\") pod \"distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg\" (UID: \"07695dfe-10c8-44af-89c5-cbb19eb57ddf\") " pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:52:05.594108 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:05.594085    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07695dfe-10c8-44af-89c5-cbb19eb57ddf-tls-certs\") pod \"distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg\" (UID: \"07695dfe-10c8-44af-89c5-cbb19eb57ddf\") " pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:52:05.678173 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:05.678097    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwdkd\" (UniqueName: \"kubernetes.io/projected/07695dfe-10c8-44af-89c5-cbb19eb57ddf-kube-api-access-pwdkd\") pod \"distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg\" (UID: \"07695dfe-10c8-44af-89c5-cbb19eb57ddf\") " pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:52:05.976337 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:05.976240    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:52:06.370065 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:06.370036    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"]
Oct 30 21:52:06.371185 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:52:06.371153    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07695dfe_10c8_44af_89c5_cbb19eb57ddf.slice/crio-ce389a910115f20b7e5fb483d603aba6c0fae19a8a2523dbb582b0afcb316ff1 WatchSource:0}: Error finding container ce389a910115f20b7e5fb483d603aba6c0fae19a8a2523dbb582b0afcb316ff1: Status 404 returned error can't find the container with id ce389a910115f20b7e5fb483d603aba6c0fae19a8a2523dbb582b0afcb316ff1
Oct 30 21:52:06.655054 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:06.654951    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" event={"ID":"07695dfe-10c8-44af-89c5-cbb19eb57ddf","Type":"ContainerStarted","Data":"cd2e69f347fc6ebeea80caac1a39009db474d9edcc0145d5bcd9da1f22c5db5a"}
Oct 30 21:52:06.655054 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:06.655026    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" event={"ID":"07695dfe-10c8-44af-89c5-cbb19eb57ddf","Type":"ContainerStarted","Data":"ce389a910115f20b7e5fb483d603aba6c0fae19a8a2523dbb582b0afcb316ff1"}
Oct 30 21:52:06.770486 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:06.770435    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podStartSLOduration=1.770417901 podStartE2EDuration="1.770417901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:06.768982975 +0000 UTC m=+1121.243486956" watchObservedRunningTime="2026-10-30 21:52:06.770417901 +0000 UTC m=+1121.244921881"
Oct 30 21:52:06.976996 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:06.976905    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:52:15.977061 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:15.977029    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:52:15.990538 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:15.990508    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:52:16.752562 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:16.752534    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:52:17.168269 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:52:17.168180    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:52:29.465739 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.465698    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"]
Oct 30 21:52:29.472221 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.472197    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:29.525632 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.525600    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-dda9f5-openshift-default-dockercfg-qx2fq\""
Oct 30 21:52:29.529999 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.529965    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"]
Oct 30 21:52:29.582524 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.582496    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istio-podinfo\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:29.582708 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.582552    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istio-data\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:29.582708 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.582643    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istiod-ca-cert\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:29.582708 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.582663    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-workload-socket\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:29.582708 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.582702    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-workload-certs\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:29.582864 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.582761    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-credential-socket\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:29.582864 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.582795    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istio-envoy\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:29.582864 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.582839    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istio-token\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:29.582957 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.582865    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8czfh\" (UniqueName: \"kubernetes.io/projected/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-kube-api-access-8czfh\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:29.684010 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.683975    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istio-envoy\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:29.684195 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.684054    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istio-token\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:29.684195 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.684091    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8czfh\" (UniqueName: \"kubernetes.io/projected/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-kube-api-access-8czfh\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:29.684195 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.684169    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istio-podinfo\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:29.684399 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.684222    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istio-data\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:29.684399 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.684269    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istiod-ca-cert\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:29.684399 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.684297    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-workload-socket\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:29.684399 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.684323    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-workload-certs\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:29.684614 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.684406    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-credential-socket\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:29.684744 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.684718    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istio-data\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:29.684818 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.684757    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-credential-socket\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:29.685044 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.685020    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-workload-socket\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:29.685136 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.685027    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-workload-certs\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:29.685562 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.685542    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istiod-ca-cert\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:29.686601 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.686582    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istio-envoy\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:29.687466 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.687440    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istio-podinfo\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:29.742655 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.742623    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8czfh\" (UniqueName: \"kubernetes.io/projected/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-kube-api-access-8czfh\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:29.742815 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.742788    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istio-token\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:29.788839 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:29.788599    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:30.227589 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:30.227555    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"]
Oct 30 21:52:30.227719 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:52:30.227630    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86ab6a2a_10d7_4957_96ac_4cc29e0fb409.slice/crio-86d8dc66fac68cbc9ee289dca70e6c85298a2fe872e5c139aadce50ee818c49f WatchSource:0}: Error finding container 86d8dc66fac68cbc9ee289dca70e6c85298a2fe872e5c139aadce50ee818c49f: Status 404 returned error can't find the container with id 86d8dc66fac68cbc9ee289dca70e6c85298a2fe872e5c139aadce50ee818c49f
Oct 30 21:52:30.230097 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:30.230067    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:52:30.230183 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:30.230129    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:52:30.230183 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:30.230157    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:52:30.854464 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:30.854428    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429" event={"ID":"86ab6a2a-10d7-4957-96ac-4cc29e0fb409","Type":"ContainerStarted","Data":"b2b6708421a1d098b8c12539dd710eb0c465c1ca04e8d912d91fb51c4715f4d0"}
Oct 30 21:52:30.854937 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:30.854492    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429" event={"ID":"86ab6a2a-10d7-4957-96ac-4cc29e0fb409","Type":"ContainerStarted","Data":"86d8dc66fac68cbc9ee289dca70e6c85298a2fe872e5c139aadce50ee818c49f"}
Oct 30 21:52:31.020176 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:31.020121    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429" podStartSLOduration=2.020106094 podStartE2EDuration="2.020106094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:31.018572504 +0000 UTC m=+1145.493076483" watchObservedRunningTime="2026-10-30 21:52:31.020106094 +0000 UTC m=+1145.494610073"
Oct 30 21:52:31.789511 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:31.789477    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:31.791321 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:31.791295    5087 patch_prober.go:29] interesting pod/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 21:52:31.791485 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:31.791365    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429" podUID="86ab6a2a-10d7-4957-96ac-4cc29e0fb409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 21:52:32.789651 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:32.789622    5087 patch_prober.go:29] interesting pod/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 21:52:32.790126 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:32.789672    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429" podUID="86ab6a2a-10d7-4957-96ac-4cc29e0fb409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 21:52:33.789614 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:33.789571    5087 patch_prober.go:29] interesting pod/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 21:52:33.789825 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:33.789642    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429" podUID="86ab6a2a-10d7-4957-96ac-4cc29e0fb409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 21:52:34.790319 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:34.790279    5087 patch_prober.go:29] interesting pod/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 21:52:34.790762 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:34.790384    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429" podUID="86ab6a2a-10d7-4957-96ac-4cc29e0fb409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 21:52:35.790114 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:35.790073    5087 patch_prober.go:29] interesting pod/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 21:52:35.790393 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:35.790143    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429" podUID="86ab6a2a-10d7-4957-96ac-4cc29e0fb409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 21:52:36.790169 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:36.790138    5087 patch_prober.go:29] interesting pod/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 30 21:52:36.790368 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:36.790197    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429" podUID="86ab6a2a-10d7-4957-96ac-4cc29e0fb409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 30 21:52:37.794538 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:37.794508    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:37.795027 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:37.794968    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:37.795716 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:37.795696    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 21:52:43.259551 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.259509    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"]
Oct 30 21:52:43.267858 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.267830    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:43.287106 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.287080    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-credential-socket\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:43.287226 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.287158    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vn2q\" (UniqueName: \"kubernetes.io/projected/b9f7c360-2a81-4e01-b104-6e9907c6cb39-kube-api-access-9vn2q\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:43.287266 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.287222    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-workload-socket\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:43.287266 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.287244    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istio-token\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:43.287266 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.287261    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-workload-certs\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:43.287426 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.287289    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istio-data\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:43.287499 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.287481    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istio-envoy\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:43.287550 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.287539    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istiod-ca-cert\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:43.287594 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.287584    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istio-podinfo\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:43.322031 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.321998    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-c79cb5-openshift-default-dockercfg-7289d\""
Oct 30 21:52:43.377307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.377275    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"]
Oct 30 21:52:43.388815 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.388778    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vn2q\" (UniqueName: \"kubernetes.io/projected/b9f7c360-2a81-4e01-b104-6e9907c6cb39-kube-api-access-9vn2q\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:43.388954 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.388849    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-workload-socket\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:43.388954 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.388882    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istio-token\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:43.388954 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.388906    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-workload-certs\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:43.389116 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.388955    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istio-data\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:43.389116 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.389026    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istio-envoy\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:43.389116 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.389072    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istiod-ca-cert\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:43.389270 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.389133    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istio-podinfo\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:43.389270 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.389184    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-credential-socket\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:43.389403 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.389302    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-workload-socket\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:43.389463 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.389399    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-workload-certs\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:43.389463 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.389451    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istio-data\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:43.389630 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.389605    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-credential-socket\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:43.389961 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.389935    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istiod-ca-cert\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:43.391438 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.391415    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istio-envoy\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:43.391923 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.391887    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istio-podinfo\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:43.446061 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.446027    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vn2q\" (UniqueName: \"kubernetes.io/projected/b9f7c360-2a81-4e01-b104-6e9907c6cb39-kube-api-access-9vn2q\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:43.446061 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.446042    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istio-token\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:43.586328 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:43.586248    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:44.019452 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:44.019410    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"]
Oct 30 21:52:44.019896 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:52:44.019873    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f7c360_2a81_4e01_b104_6e9907c6cb39.slice/crio-89d610f4ccf183d28c0be9fba8d67b4d4069171716e7de2d01ee159d8f654b77 WatchSource:0}: Error finding container 89d610f4ccf183d28c0be9fba8d67b4d4069171716e7de2d01ee159d8f654b77: Status 404 returned error can't find the container with id 89d610f4ccf183d28c0be9fba8d67b4d4069171716e7de2d01ee159d8f654b77
Oct 30 21:52:44.022239 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:44.022208    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:52:44.022308 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:44.022275    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:52:44.022308 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:44.022302    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:52:44.973870 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:44.973833    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l" event={"ID":"b9f7c360-2a81-4e01-b104-6e9907c6cb39","Type":"ContainerStarted","Data":"425a4c700a055308a86b8dd186fbb8ae7514dfc3e3c1c17713e941a2f54c5aaf"}
Oct 30 21:52:44.974303 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:44.973894    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l" event={"ID":"b9f7c360-2a81-4e01-b104-6e9907c6cb39","Type":"ContainerStarted","Data":"89d610f4ccf183d28c0be9fba8d67b4d4069171716e7de2d01ee159d8f654b77"}
Oct 30 21:52:45.194666 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:45.194608    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l" podStartSLOduration=2.194588626 podStartE2EDuration="2.194588626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:45.141934014 +0000 UTC m=+1159.616437997" watchObservedRunningTime="2026-10-30 21:52:45.194588626 +0000 UTC m=+1159.669092671"
Oct 30 21:52:45.203826 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:45.203802    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"]
Oct 30 21:52:45.204003 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:45.203984    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm" podUID="a4279c46-56a1-4788-b768-76b46d2988db" containerName="istio-proxy" containerID="cri-o://b7357cc279ea07362f500ecb30445763139d014bf50092c6c5b6f12d850d19c1" gracePeriod=30
Oct 30 21:52:45.587307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:45.587262    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:45.589139 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:45.589109    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 21:52:45.589292 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:45.589170    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l" podUID="b9f7c360-2a81-4e01-b104-6e9907c6cb39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 21:52:46.587666 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:46.587630    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 21:52:46.588051 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:46.587689    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l" podUID="b9f7c360-2a81-4e01-b104-6e9907c6cb39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 21:52:47.187025 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:52:47.186985    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:52:47.587037 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:47.586998    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 21:52:47.587230 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:47.587066    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l" podUID="b9f7c360-2a81-4e01-b104-6e9907c6cb39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 21:52:48.586899 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:48.586864    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 21:52:48.587546 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:48.587503    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l" podUID="b9f7c360-2a81-4e01-b104-6e9907c6cb39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 21:52:49.586760 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:49.586715    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 21:52:49.586970 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:49.586791    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l" podUID="b9f7c360-2a81-4e01-b104-6e9907c6cb39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 21:52:49.815164 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:49.815115    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-4k5dm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 21:52:49.815372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:49.815202    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm" podUID="a4279c46-56a1-4788-b768-76b46d2988db" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 21:52:50.476293 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:50.476240    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:52:50.587129 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:50.587092    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 21:52:50.587561 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:50.587146    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l" podUID="b9f7c360-2a81-4e01-b104-6e9907c6cb39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 21:52:50.600523 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:50.600495    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4279c46-56a1-4788-b768-76b46d2988db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4279c46-56a1-4788-b768-76b46d2988db-istio-podinfo\") pod \"a4279c46-56a1-4788-b768-76b46d2988db\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") "
Oct 30 21:52:50.600693 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:50.600556    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-credential-socket\") pod \"a4279c46-56a1-4788-b768-76b46d2988db\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") "
Oct 30 21:52:50.600693 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:50.600596    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-istio-envoy\") pod \"a4279c46-56a1-4788-b768-76b46d2988db\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") "
Oct 30 21:52:50.600811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:50.600694    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4279c46-56a1-4788-b768-76b46d2988db-istio-token\" (UniqueName: \"kubernetes.io/projected/a4279c46-56a1-4788-b768-76b46d2988db-istio-token\") pod \"a4279c46-56a1-4788-b768-76b46d2988db\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") "
Oct 30 21:52:50.600811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:50.600727    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-workload-certs\") pod \"a4279c46-56a1-4788-b768-76b46d2988db\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") "
Oct 30 21:52:50.600914 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:50.600807    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4279c46-56a1-4788-b768-76b46d2988db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4279c46-56a1-4788-b768-76b46d2988db-istiod-ca-cert\") pod \"a4279c46-56a1-4788-b768-76b46d2988db\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") "
Oct 30 21:52:50.600914 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:50.600869    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-workload-socket\") pod \"a4279c46-56a1-4788-b768-76b46d2988db\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") "
Oct 30 21:52:50.600914 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:50.600894    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-credential-socket" pod "a4279c46-56a1-4788-b768-76b46d2988db" (UID: "a4279c46-56a1-4788-b768-76b46d2988db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:52:50.601063 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:50.600925    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-istio-data\") pod \"a4279c46-56a1-4788-b768-76b46d2988db\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") "
Oct 30 21:52:50.601063 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:50.600984    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4279c46-56a1-4788-b768-76b46d2988db-kube-api-access-phsgn\" (UniqueName: \"kubernetes.io/projected/a4279c46-56a1-4788-b768-76b46d2988db-kube-api-access-phsgn\") pod \"a4279c46-56a1-4788-b768-76b46d2988db\" (UID: \"a4279c46-56a1-4788-b768-76b46d2988db\") "
Oct 30 21:52:50.601157 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:50.601125    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-workload-certs" pod "a4279c46-56a1-4788-b768-76b46d2988db" (UID: "a4279c46-56a1-4788-b768-76b46d2988db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:52:50.601542 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:50.601518    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:52:50.601780 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:50.601763    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:52:50.602902 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:50.602195    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-istio-data" pod "a4279c46-56a1-4788-b768-76b46d2988db" (UID: "a4279c46-56a1-4788-b768-76b46d2988db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:52:50.603097 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:50.603061    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-workload-socket" pod "a4279c46-56a1-4788-b768-76b46d2988db" (UID: "a4279c46-56a1-4788-b768-76b46d2988db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:52:50.603412 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:50.603387    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4279c46-56a1-4788-b768-76b46d2988db-istiod-ca-cert" pod "a4279c46-56a1-4788-b768-76b46d2988db" (UID: "a4279c46-56a1-4788-b768-76b46d2988db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:52:50.603932 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:50.603911    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4279c46-56a1-4788-b768-76b46d2988db-kube-api-access-phsgn" pod "a4279c46-56a1-4788-b768-76b46d2988db" (UID: "a4279c46-56a1-4788-b768-76b46d2988db"). InnerVolumeSpecName "kube-api-access-phsgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:50.604596 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:50.604574    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4279c46-56a1-4788-b768-76b46d2988db-istio-token" pod "a4279c46-56a1-4788-b768-76b46d2988db" (UID: "a4279c46-56a1-4788-b768-76b46d2988db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:52:50.609851 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:50.609816    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4279c46-56a1-4788-b768-76b46d2988db-istio-podinfo" pod "a4279c46-56a1-4788-b768-76b46d2988db" (UID: "a4279c46-56a1-4788-b768-76b46d2988db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 21:52:50.613706 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:50.613676    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-istio-envoy" pod "a4279c46-56a1-4788-b768-76b46d2988db" (UID: "a4279c46-56a1-4788-b768-76b46d2988db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:52:50.702587 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:50.702554    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4279c46-56a1-4788-b768-76b46d2988db-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:52:50.702587 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:50.702580    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:52:50.702587 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:50.702591    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:52:50.702903 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:50.702601    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phsgn\" (UniqueName: \"kubernetes.io/projected/a4279c46-56a1-4788-b768-76b46d2988db-kube-api-access-phsgn\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:52:50.702903 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:50.702610    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4279c46-56a1-4788-b768-76b46d2988db-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:52:50.702903 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:50.702620    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4279c46-56a1-4788-b768-76b46d2988db-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:52:50.702903 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:50.702627    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4279c46-56a1-4788-b768-76b46d2988db-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:52:51.026880 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:51.026848    5087 generic.go:356] "Generic (PLEG): container finished" podID="a4279c46-56a1-4788-b768-76b46d2988db" containerID="b7357cc279ea07362f500ecb30445763139d014bf50092c6c5b6f12d850d19c1" exitCode=0
Oct 30 21:52:51.027062 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:51.026907    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"
Oct 30 21:52:51.027062 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:51.026943    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm" event={"ID":"a4279c46-56a1-4788-b768-76b46d2988db","Type":"ContainerDied","Data":"b7357cc279ea07362f500ecb30445763139d014bf50092c6c5b6f12d850d19c1"}
Oct 30 21:52:51.027062 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:51.027018    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm" event={"ID":"a4279c46-56a1-4788-b768-76b46d2988db","Type":"ContainerDied","Data":"24767bb4ed9b7d7dd640685d954156d36b265673914bf302a9f3c81f51a1c7aa"}
Oct 30 21:52:51.027062 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:51.027053    5087 scope.go:122] "RemoveContainer" containerID="b7357cc279ea07362f500ecb30445763139d014bf50092c6c5b6f12d850d19c1"
Oct 30 21:52:51.041134 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:51.041108    5087 scope.go:122] "RemoveContainer" containerID="b7357cc279ea07362f500ecb30445763139d014bf50092c6c5b6f12d850d19c1"
Oct 30 21:52:51.041516 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:52:51.041496    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7357cc279ea07362f500ecb30445763139d014bf50092c6c5b6f12d850d19c1\": container with ID starting with b7357cc279ea07362f500ecb30445763139d014bf50092c6c5b6f12d850d19c1 not found: ID does not exist" containerID="b7357cc279ea07362f500ecb30445763139d014bf50092c6c5b6f12d850d19c1"
Oct 30 21:52:51.041627 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:51.041523    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7357cc279ea07362f500ecb30445763139d014bf50092c6c5b6f12d850d19c1"} err="failed to get container status \"b7357cc279ea07362f500ecb30445763139d014bf50092c6c5b6f12d850d19c1\": rpc error: code = NotFound desc = could not find container \"b7357cc279ea07362f500ecb30445763139d014bf50092c6c5b6f12d850d19c1\": container with ID starting with b7357cc279ea07362f500ecb30445763139d014bf50092c6c5b6f12d850d19c1 not found: ID does not exist"
Oct 30 21:52:51.193013 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:51.192979    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"]
Oct 30 21:52:51.243204 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:51.243175    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-4k5dm"]
Oct 30 21:52:51.591031 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:51.591002    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:51.591459 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:51.591451    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:51.592425 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:51.592403    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 21:52:51.710480 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:51.710447    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4279c46-56a1-4788-b768-76b46d2988db" path="/var/lib/kubelet/pods/a4279c46-56a1-4788-b768-76b46d2988db/volumes"
Oct 30 21:52:59.087176 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.087143    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"]
Oct 30 21:52:59.087921 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.087900    5087 cpu_manager.go:414] "containerMap: removing container" podUID="a4279c46-56a1-4788-b768-76b46d2988db" containerName="istio-proxy"
Oct 30 21:52:59.088048 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.087923    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4279c46-56a1-4788-b768-76b46d2988db" containerName="istio-proxy"
Oct 30 21:52:59.088110 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.088071    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4279c46-56a1-4788-b768-76b46d2988db" containerName="istio-proxy"
Oct 30 21:52:59.093273 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.093253    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:52:59.156422 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.156391    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"]
Oct 30 21:52:59.210265 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.210231    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8w7c\" (UniqueName: \"kubernetes.io/projected/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-kube-api-access-w8w7c\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:52:59.210473 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.210284    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istio-token\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:52:59.210473 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.210388    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-workload-certs\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:52:59.210473 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.210447    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istio-envoy\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:52:59.210473 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.210470    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istio-podinfo\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:52:59.210628 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.210492    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-credential-socket\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:52:59.210674 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.210625    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-workload-socket\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:52:59.210674 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.210669    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istio-data\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:52:59.210799 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.210721    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istiod-ca-cert\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:52:59.311849 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.311785    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8w7c\" (UniqueName: \"kubernetes.io/projected/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-kube-api-access-w8w7c\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:52:59.312048 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.311891    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istio-token\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:52:59.312048 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.311939    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-workload-certs\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:52:59.312048 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.311974    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istio-envoy\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:52:59.312048 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.311996    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istio-podinfo\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:52:59.312048 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.312022    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-credential-socket\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:52:59.312325 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.312053    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-workload-socket\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:52:59.312325 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.312079    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istio-data\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:52:59.312325 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.312106    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istiod-ca-cert\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:52:59.313071 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.313006    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istiod-ca-cert\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:52:59.313558 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.313173    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-workload-certs\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:52:59.313558 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.313302    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-credential-socket\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:52:59.313862 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.313568    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-workload-socket\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:52:59.313862 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.313738    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istio-data\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:52:59.315051 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.315032    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istio-podinfo\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:52:59.316000 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.315985    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istio-envoy\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:52:59.369284 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.369216    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istio-token\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:52:59.369429 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.369335    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8w7c\" (UniqueName: \"kubernetes.io/projected/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-kube-api-access-w8w7c\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:52:59.407412 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.407386    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:52:59.837249 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.837207    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"]
Oct 30 21:52:59.875800 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:52:59.875763    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf95e4084_7a0f_4fa3_ad48_e34ee3eab117.slice/crio-20d196bdf030d33d8d46abdce089cf495e14018e890238cf9003fe34ca2b2896 WatchSource:0}: Error finding container 20d196bdf030d33d8d46abdce089cf495e14018e890238cf9003fe34ca2b2896: Status 404 returned error can't find the container with id 20d196bdf030d33d8d46abdce089cf495e14018e890238cf9003fe34ca2b2896
Oct 30 21:52:59.878588 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.878554    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:52:59.878700 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.878635    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:52:59.878700 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:52:59.878678    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:53:00.128736 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:00.128636    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc" event={"ID":"f95e4084-7a0f-4fa3-ad48-e34ee3eab117","Type":"ContainerStarted","Data":"520272e544a5110764fcfce3dbc31ba6560ecb5be68a621470878be98d087f74"}
Oct 30 21:53:00.129211 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:00.128747    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc" event={"ID":"f95e4084-7a0f-4fa3-ad48-e34ee3eab117","Type":"ContainerStarted","Data":"20d196bdf030d33d8d46abdce089cf495e14018e890238cf9003fe34ca2b2896"}
Oct 30 21:53:00.261170 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:00.261100    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc" podStartSLOduration=1.261080088 podStartE2EDuration="1.261080088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:00.253964144 +0000 UTC m=+1174.728468122" watchObservedRunningTime="2026-10-30 21:53:00.261080088 +0000 UTC m=+1174.735584068"
Oct 30 21:53:00.408121 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:00.408039    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:53:00.409484 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:00.409449    5087 patch_prober.go:29] interesting pod/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 21:53:00.409641 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:00.409504    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc" podUID="f95e4084-7a0f-4fa3-ad48-e34ee3eab117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 21:53:01.408108 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:01.408078    5087 patch_prober.go:29] interesting pod/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 21:53:01.408661 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:01.408137    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc" podUID="f95e4084-7a0f-4fa3-ad48-e34ee3eab117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 21:53:02.408125 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:02.408088    5087 patch_prober.go:29] interesting pod/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 21:53:02.408656 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:02.408155    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc" podUID="f95e4084-7a0f-4fa3-ad48-e34ee3eab117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 21:53:03.408590 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:03.408552    5087 patch_prober.go:29] interesting pod/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 21:53:03.408959 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:03.408616    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc" podUID="f95e4084-7a0f-4fa3-ad48-e34ee3eab117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 21:53:04.407813 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:04.407777    5087 patch_prober.go:29] interesting pod/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 21:53:04.407995 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:04.407840    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc" podUID="f95e4084-7a0f-4fa3-ad48-e34ee3eab117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 21:53:05.408337 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:05.408284    5087 patch_prober.go:29] interesting pod/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 21:53:05.408891 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:05.408387    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc" podUID="f95e4084-7a0f-4fa3-ad48-e34ee3eab117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 21:53:06.408589 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:06.408545    5087 patch_prober.go:29] interesting pod/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body=
Oct 30 21:53:06.408986 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:06.408619    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc" podUID="f95e4084-7a0f-4fa3-ad48-e34ee3eab117" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused"
Oct 30 21:53:07.414049 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:07.414013    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:53:07.414558 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:07.414539    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:53:07.415137 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:07.415118    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 21:53:12.932306 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:12.932269    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"]
Oct 30 21:53:12.936374 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:12.936336    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:13.000767 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.000738    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-istio-envoy\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:13.000928 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.000777    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2fda28b-f446-4917-bbe1-721ecff56fd8-istio-token\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:13.000928 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.000797    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2fda28b-f446-4917-bbe1-721ecff56fd8-istiod-ca-cert\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:13.001027 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.000926    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-workload-socket\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:13.001087 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.001069    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2fda28b-f446-4917-bbe1-721ecff56fd8-istio-podinfo\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:13.001139 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.001123    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-istio-data\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:13.001261 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.001240    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhp4x\" (UniqueName: \"kubernetes.io/projected/b2fda28b-f446-4917-bbe1-721ecff56fd8-kube-api-access-dhp4x\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:13.001327 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.001277    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-workload-certs\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:13.001449 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.001424    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-credential-socket\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:13.102693 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.102658    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-credential-socket\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:13.102893 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.102729    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-istio-envoy\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:13.102893 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.102755    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2fda28b-f446-4917-bbe1-721ecff56fd8-istio-token\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:13.102893 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.102798    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2fda28b-f446-4917-bbe1-721ecff56fd8-istiod-ca-cert\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:13.102893 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.102843    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-workload-socket\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:13.103132 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.102914    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2fda28b-f446-4917-bbe1-721ecff56fd8-istio-podinfo\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:13.103132 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.102957    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-istio-data\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:13.103132 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.103022    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhp4x\" (UniqueName: \"kubernetes.io/projected/b2fda28b-f446-4917-bbe1-721ecff56fd8-kube-api-access-dhp4x\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:13.103132 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.103049    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-workload-certs\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:13.103132 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.103105    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-credential-socket\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:13.103412 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.103395    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-workload-certs\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:13.103469 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.103456    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-istio-data\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:13.103719 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.103692    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-workload-socket\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:13.104106 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.104081    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2fda28b-f446-4917-bbe1-721ecff56fd8-istiod-ca-cert\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:13.105287 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.105263    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-istio-envoy\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:13.106370 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.106333    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2fda28b-f446-4917-bbe1-721ecff56fd8-istio-podinfo\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:13.141078 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.141053    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"]
Oct 30 21:53:13.159700 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.159666    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2fda28b-f446-4917-bbe1-721ecff56fd8-istio-token\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:13.172590 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.172564    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhp4x\" (UniqueName: \"kubernetes.io/projected/b2fda28b-f446-4917-bbe1-721ecff56fd8-kube-api-access-dhp4x\") pod \"e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") " pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:13.251137 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.251106    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:13.681448 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.681371    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"]
Oct 30 21:53:13.718176 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:53:13.718145    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2fda28b_f446_4917_bbe1_721ecff56fd8.slice/crio-7a688b283a4064c482532cc09e22a8d5593cbad2e02ba1652a5e4722dcb4739f WatchSource:0}: Error finding container 7a688b283a4064c482532cc09e22a8d5593cbad2e02ba1652a5e4722dcb4739f: Status 404 returned error can't find the container with id 7a688b283a4064c482532cc09e22a8d5593cbad2e02ba1652a5e4722dcb4739f
Oct 30 21:53:13.720296 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.720263    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:53:13.720400 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.720337    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:53:13.720400 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:13.720390    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:53:14.271372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:14.269665    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf" event={"ID":"b2fda28b-f446-4917-bbe1-721ecff56fd8","Type":"ContainerStarted","Data":"2c1017b716a5455a38caaacb432a5cc3ac6b0f143a3e7fdc0fd853888137021b"}
Oct 30 21:53:14.271372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:14.269752    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf" event={"ID":"b2fda28b-f446-4917-bbe1-721ecff56fd8","Type":"ContainerStarted","Data":"7a688b283a4064c482532cc09e22a8d5593cbad2e02ba1652a5e4722dcb4739f"}
Oct 30 21:53:14.428978 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:14.428927    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf" podStartSLOduration=2.428911959 podStartE2EDuration="2.428911959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:14.426112424 +0000 UTC m=+1188.900616421" watchObservedRunningTime="2026-10-30 21:53:14.428911959 +0000 UTC m=+1188.903415938"
Oct 30 21:53:15.261571 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:15.261542    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:15.263114 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:15.263089    5087 patch_prober.go:29] interesting pod/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 21:53:15.263251 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:15.263146    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf" podUID="b2fda28b-f446-4917-bbe1-721ecff56fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 21:53:15.280214 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:15.280188    5087 generic.go:356] "Generic (PLEG): container finished" podID="f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49" containerID="b9516405757309c69ef0c7f3e9408631317a07932c3b11c0e49661dbe847c67e" exitCode=0
Oct 30 21:53:15.280650 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:15.280269    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jlrnh-qhkzt" event={"ID":"f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49","Type":"ContainerDied","Data":"b9516405757309c69ef0c7f3e9408631317a07932c3b11c0e49661dbe847c67e"}
Oct 30 21:53:16.260370 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:16.259755    5087 patch_prober.go:29] interesting pod/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 21:53:16.260370 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:16.259813    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf" podUID="b2fda28b-f446-4917-bbe1-721ecff56fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 21:53:16.480319 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:16.480290    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jlrnh-qhkzt"
Oct 30 21:53:16.569091 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:16.569001    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49-trusted-ca-bundle\") pod \"f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49\" (UID: \"f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49\") "
Oct 30 21:53:16.569091 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:16.569049    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49-archives-path\") pod \"f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49\" (UID: \"f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49\") "
Oct 30 21:53:16.569333 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:16.569094    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49-kube-api-access-m2h9p\" (UniqueName: \"kubernetes.io/projected/f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49-kube-api-access-m2h9p\") pod \"f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49\" (UID: \"f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49\") "
Oct 30 21:53:16.569333 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:16.569137    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49-service-ca-bundle\") pod \"f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49\" (UID: \"f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49\") "
Oct 30 21:53:16.569480 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:16.569444    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49-trusted-ca-bundle" pod "f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49" (UID: "f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:16.569528 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:16.569472    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49-archives-path" pod "f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49" (UID: "f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:53:16.569573 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:16.569538    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49-service-ca-bundle" pod "f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49" (UID: "f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:16.569741 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:16.569723    5087 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49-trusted-ca-bundle\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:16.569781 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:16.569747    5087 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49-archives-path\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:16.569781 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:16.569762    5087 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49-service-ca-bundle\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:16.571417 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:16.571389    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49-kube-api-access-m2h9p" pod "f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49" (UID: "f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49"). InnerVolumeSpecName "kube-api-access-m2h9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:16.671214 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:16.671174    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2h9p\" (UniqueName: \"kubernetes.io/projected/f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49-kube-api-access-m2h9p\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:17.189619 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:53:17.189570    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:53:17.259788 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:17.259753    5087 patch_prober.go:29] interesting pod/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 21:53:17.260050 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:17.260025    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf" podUID="b2fda28b-f446-4917-bbe1-721ecff56fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 21:53:17.296022 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:17.295986    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jlrnh-qhkzt" event={"ID":"f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49","Type":"ContainerDied","Data":"b12ddd34de77f0dd068a47f65cd48aa381cbe11a4ccf7af9009cbdb8d8b03662"}
Oct 30 21:53:17.296022 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:17.296024    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jlrnh-qhkzt"
Oct 30 21:53:17.296261 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:17.296035    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b12ddd34de77f0dd068a47f65cd48aa381cbe11a4ccf7af9009cbdb8d8b03662"
Oct 30 21:53:18.253746 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:18.253703    5087 patch_prober.go:29] interesting pod/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 21:53:18.254406 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:18.253769    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf" podUID="b2fda28b-f446-4917-bbe1-721ecff56fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 21:53:19.264373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:19.258717    5087 patch_prober.go:29] interesting pod/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 21:53:19.264373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:19.258779    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf" podUID="b2fda28b-f446-4917-bbe1-721ecff56fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 21:53:20.252264 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:20.252226    5087 patch_prober.go:29] interesting pod/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Oct 30 21:53:20.252482 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:20.252297    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf" podUID="b2fda28b-f446-4917-bbe1-721ecff56fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Oct 30 21:53:21.256428 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:21.256397    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:21.256914 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:21.256877    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:21.257541 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:21.257520    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 21:53:25.855559 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:25.855530    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 21:53:25.861250 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:25.861227    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 21:53:25.865445 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:25.865425    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 21:53:25.870938 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:25.870918    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 21:53:47.169495 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:53:47.169412    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:53:47.893874 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:47.893842    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c04fcec-4dng2"]
Oct 30 21:53:47.894477 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:47.894458    5087 cpu_manager.go:414] "containerMap: removing container" podUID="f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49" containerName="insights-gathering"
Oct 30 21:53:47.894583 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:47.894478    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49" containerName="insights-gathering"
Oct 30 21:53:47.894647 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:47.894613    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="f44dd57f-7e32-4ccd-b2d7-aeb7bb923c49" containerName="insights-gathering"
Oct 30 21:53:47.898322 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:47.898301    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c04fcec-4dng2"
Oct 30 21:53:47.951835 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:47.951715    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:53:48.001217 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:48.001181    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:53:48.007401 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:48.007375    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c04fcec-4dng2"]
Oct 30 21:53:48.056575 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:48.056546    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20fc6591-9453-4da6-977b-e30509b13264-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c04fcec-4dng2\" (UID: \"20fc6591-9453-4da6-977b-e30509b13264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c04fcec-4dng2"
Oct 30 21:53:48.056729 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:48.056615    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20fc6591-9453-4da6-977b-e30509b13264-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c04fcec-4dng2\" (UID: \"20fc6591-9453-4da6-977b-e30509b13264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c04fcec-4dng2"
Oct 30 21:53:48.056819 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:48.056794    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x79s\" (UniqueName: \"kubernetes.io/projected/20fc6591-9453-4da6-977b-e30509b13264-kube-api-access-2x79s\") pod \"maas-api-revoke-sub-9c04fcec-4dng2\" (UID: \"20fc6591-9453-4da6-977b-e30509b13264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c04fcec-4dng2"
Oct 30 21:53:48.157736 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:48.157651    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20fc6591-9453-4da6-977b-e30509b13264-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c04fcec-4dng2\" (UID: \"20fc6591-9453-4da6-977b-e30509b13264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c04fcec-4dng2"
Oct 30 21:53:48.157901 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:48.157754    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20fc6591-9453-4da6-977b-e30509b13264-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c04fcec-4dng2\" (UID: \"20fc6591-9453-4da6-977b-e30509b13264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c04fcec-4dng2"
Oct 30 21:53:48.157901 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:48.157835    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x79s\" (UniqueName: \"kubernetes.io/projected/20fc6591-9453-4da6-977b-e30509b13264-kube-api-access-2x79s\") pod \"maas-api-revoke-sub-9c04fcec-4dng2\" (UID: \"20fc6591-9453-4da6-977b-e30509b13264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c04fcec-4dng2"
Oct 30 21:53:48.158516 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:48.158495    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20fc6591-9453-4da6-977b-e30509b13264-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c04fcec-4dng2\" (UID: \"20fc6591-9453-4da6-977b-e30509b13264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c04fcec-4dng2"
Oct 30 21:53:48.160327 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:48.160303    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20fc6591-9453-4da6-977b-e30509b13264-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c04fcec-4dng2\" (UID: \"20fc6591-9453-4da6-977b-e30509b13264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c04fcec-4dng2"
Oct 30 21:53:48.215183 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:48.215153    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x79s\" (UniqueName: \"kubernetes.io/projected/20fc6591-9453-4da6-977b-e30509b13264-kube-api-access-2x79s\") pod \"maas-api-revoke-sub-9c04fcec-4dng2\" (UID: \"20fc6591-9453-4da6-977b-e30509b13264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c04fcec-4dng2"
Oct 30 21:53:48.512732 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:48.512696    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c04fcec-4dng2"
Oct 30 21:53:48.893324 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:48.893290    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c04fcec-4dng2"]
Oct 30 21:53:48.893767 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:53:48.893727    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20fc6591_9453_4da6_977b_e30509b13264.slice/crio-dd9909e4f974e2ae825e0b9be990675e5deaa856f07c1f0c14f8f3ec264816f3 WatchSource:0}: Error finding container dd9909e4f974e2ae825e0b9be990675e5deaa856f07c1f0c14f8f3ec264816f3: Status 404 returned error can't find the container with id dd9909e4f974e2ae825e0b9be990675e5deaa856f07c1f0c14f8f3ec264816f3
Oct 30 21:53:49.521285 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:49.521253    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c04fcec-4dng2" event={"ID":"20fc6591-9453-4da6-977b-e30509b13264","Type":"ContainerStarted","Data":"fb1bfb3ffb0ca132e02cec33dec62beb79875a620a46a8d55eaf80d567ce0b4a"}
Oct 30 21:53:49.521878 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:49.521304    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c04fcec-4dng2" event={"ID":"20fc6591-9453-4da6-977b-e30509b13264","Type":"ContainerStarted","Data":"dd9909e4f974e2ae825e0b9be990675e5deaa856f07c1f0c14f8f3ec264816f3"}
Oct 30 21:53:49.641227 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:49.641174    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c04fcec-4dng2" podStartSLOduration=2.641161483 podStartE2EDuration="2.641161483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:49.640574025 +0000 UTC m=+1224.115078005" watchObservedRunningTime="2026-10-30 21:53:49.641161483 +0000 UTC m=+1224.115665462"
Oct 30 21:53:50.529271 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:50.529240    5087 generic.go:356] "Generic (PLEG): container finished" podID="20fc6591-9453-4da6-977b-e30509b13264" containerID="fb1bfb3ffb0ca132e02cec33dec62beb79875a620a46a8d55eaf80d567ce0b4a" exitCode=0
Oct 30 21:53:50.529704 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:50.529286    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c04fcec-4dng2" event={"ID":"20fc6591-9453-4da6-977b-e30509b13264","Type":"ContainerDied","Data":"fb1bfb3ffb0ca132e02cec33dec62beb79875a620a46a8d55eaf80d567ce0b4a"}
Oct 30 21:53:51.720011 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:51.719984    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c04fcec-4dng2"
Oct 30 21:53:51.807904 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:51.807820    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20fc6591-9453-4da6-977b-e30509b13264-kube-api-access-2x79s\" (UniqueName: \"kubernetes.io/projected/20fc6591-9453-4da6-977b-e30509b13264-kube-api-access-2x79s\") pod \"20fc6591-9453-4da6-977b-e30509b13264\" (UID: \"20fc6591-9453-4da6-977b-e30509b13264\") "
Oct 30 21:53:51.808075 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:51.807937    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20fc6591-9453-4da6-977b-e30509b13264-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20fc6591-9453-4da6-977b-e30509b13264-maas-api-openshift-ca\") pod \"20fc6591-9453-4da6-977b-e30509b13264\" (UID: \"20fc6591-9453-4da6-977b-e30509b13264\") "
Oct 30 21:53:51.808075 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:51.808044    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20fc6591-9453-4da6-977b-e30509b13264-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20fc6591-9453-4da6-977b-e30509b13264-maas-api-xks-ca\") pod \"20fc6591-9453-4da6-977b-e30509b13264\" (UID: \"20fc6591-9453-4da6-977b-e30509b13264\") "
Oct 30 21:53:51.808310 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:51.808282    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20fc6591-9453-4da6-977b-e30509b13264-maas-api-openshift-ca" pod "20fc6591-9453-4da6-977b-e30509b13264" (UID: "20fc6591-9453-4da6-977b-e30509b13264"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:51.808901 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:51.808554    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20fc6591-9453-4da6-977b-e30509b13264-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:51.810145 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:51.810122    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20fc6591-9453-4da6-977b-e30509b13264-kube-api-access-2x79s" pod "20fc6591-9453-4da6-977b-e30509b13264" (UID: "20fc6591-9453-4da6-977b-e30509b13264"). InnerVolumeSpecName "kube-api-access-2x79s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:51.810273 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:51.810221    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20fc6591-9453-4da6-977b-e30509b13264-maas-api-xks-ca" pod "20fc6591-9453-4da6-977b-e30509b13264" (UID: "20fc6591-9453-4da6-977b-e30509b13264"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:53:51.909777 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:51.909742    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x79s\" (UniqueName: \"kubernetes.io/projected/20fc6591-9453-4da6-977b-e30509b13264-kube-api-access-2x79s\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:51.909777 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:51.909775    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20fc6591-9453-4da6-977b-e30509b13264-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:53:52.548056 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:52.548029    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c04fcec-4dng2"
Oct 30 21:53:52.548270 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:52.548074    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c04fcec-4dng2" event={"ID":"20fc6591-9453-4da6-977b-e30509b13264","Type":"ContainerDied","Data":"dd9909e4f974e2ae825e0b9be990675e5deaa856f07c1f0c14f8f3ec264816f3"}
Oct 30 21:53:52.548270 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:53:52.548122    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd9909e4f974e2ae825e0b9be990675e5deaa856f07c1f0c14f8f3ec264816f3"
Oct 30 21:54:08.061689 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:08.061649    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad56773-dl4px"]
Oct 30 21:54:08.108498 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:08.108467    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bad56773-dl4px"]
Oct 30 21:54:09.714437 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:09.714400    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be908977-826e-4686-a7bf-035a233afac5" path="/var/lib/kubelet/pods/be908977-826e-4686-a7bf-035a233afac5/volumes"
Oct 30 21:54:14.919611 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:14.919578    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"]
Oct 30 21:54:14.920129 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:14.920115    5087 cpu_manager.go:414] "containerMap: removing container" podUID="20fc6591-9453-4da6-977b-e30509b13264" containerName="revoke-keys"
Oct 30 21:54:14.920172 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:14.920130    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20fc6591-9453-4da6-977b-e30509b13264" containerName="revoke-keys"
Oct 30 21:54:14.920222 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:14.920214    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="20fc6591-9453-4da6-977b-e30509b13264" containerName="revoke-keys"
Oct 30 21:54:14.923694 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:14.923674    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:15.073863 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.073827    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2tf8\" (UniqueName: \"kubernetes.io/projected/faa7e534-c66d-4670-83cd-c170bcc355d6-kube-api-access-z2tf8\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:15.074067 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.073899    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-istio-data\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:15.074067 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.073941    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa7e534-c66d-4670-83cd-c170bcc355d6-istio-podinfo\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:15.074067 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.073985    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-workload-certs\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:15.074067 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.074037    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-credential-socket\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:15.074294 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.074091    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa7e534-c66d-4670-83cd-c170bcc355d6-istiod-ca-cert\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:15.074294 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.074125    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-workload-socket\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:15.074294 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.074148    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa7e534-c66d-4670-83cd-c170bcc355d6-istio-token\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:15.074294 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.074212    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-istio-envoy\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:15.159283 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.159253    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"]
Oct 30 21:54:15.175381 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.175282    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-istio-data\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:15.175381 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.175329    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa7e534-c66d-4670-83cd-c170bcc355d6-istio-podinfo\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:15.175558 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.175513    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-workload-certs\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:15.175621 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.175600    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-credential-socket\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:15.175678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.175666    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa7e534-c66d-4670-83cd-c170bcc355d6-istiod-ca-cert\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:15.175738 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.175702    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-workload-socket\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:15.175738 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.175727    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa7e534-c66d-4670-83cd-c170bcc355d6-istio-token\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:15.175852 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.175792    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-istio-envoy\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:15.175852 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.175824    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-istio-data\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:15.175954 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.175865    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-workload-certs\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:15.175954 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.175908    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-credential-socket\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:15.175954 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.175921    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2tf8\" (UniqueName: \"kubernetes.io/projected/faa7e534-c66d-4670-83cd-c170bcc355d6-kube-api-access-z2tf8\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:15.176117 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.176070    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-workload-socket\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:15.176613 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.176585    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa7e534-c66d-4670-83cd-c170bcc355d6-istiod-ca-cert\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:15.178231 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.178208    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-istio-envoy\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:15.178415 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.178399    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa7e534-c66d-4670-83cd-c170bcc355d6-istio-podinfo\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:15.233778 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.233752    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa7e534-c66d-4670-83cd-c170bcc355d6-istio-token\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:15.233778 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.233774    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2tf8\" (UniqueName: \"kubernetes.io/projected/faa7e534-c66d-4670-83cd-c170bcc355d6-kube-api-access-z2tf8\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:15.236775 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.236756    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:15.675466 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:54:15.675432    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa7e534_c66d_4670_83cd_c170bcc355d6.slice/crio-2712e51e68b7262d214a79f8a5cb3c4419930de1c651de2502aa0b0d031cf80c WatchSource:0}: Error finding container 2712e51e68b7262d214a79f8a5cb3c4419930de1c651de2502aa0b0d031cf80c: Status 404 returned error can't find the container with id 2712e51e68b7262d214a79f8a5cb3c4419930de1c651de2502aa0b0d031cf80c
Oct 30 21:54:15.677119 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.677006    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"]
Oct 30 21:54:15.678153 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.678121    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:54:15.678215 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.678199    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:54:15.678260 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.678236    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:54:15.720477 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:15.720449    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w" event={"ID":"faa7e534-c66d-4670-83cd-c170bcc355d6","Type":"ContainerStarted","Data":"2712e51e68b7262d214a79f8a5cb3c4419930de1c651de2502aa0b0d031cf80c"}
Oct 30 21:54:16.728717 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:16.728681    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w" event={"ID":"faa7e534-c66d-4670-83cd-c170bcc355d6","Type":"ContainerStarted","Data":"a913b4148c77ed1adce59741beac8509d17d1e9fba2075b195ecdd13ff0b4933"}
Oct 30 21:54:16.891270 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:16.891218    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w" podStartSLOduration=2.8912042810000003 podStartE2EDuration="2.891204281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:16.888326138 +0000 UTC m=+1251.362830116" watchObservedRunningTime="2026-10-30 21:54:16.891204281 +0000 UTC m=+1251.365708260"
Oct 30 21:54:17.195976 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:54:17.195935    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:54:17.240077 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:17.240047    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:17.240410 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:17.240387    5087 patch_prober.go:29] interesting pod/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 21:54:17.240581 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:17.240558    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w" podUID="faa7e534-c66d-4670-83cd-c170bcc355d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 21:54:18.240751 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:18.240713    5087 patch_prober.go:29] interesting pod/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 21:54:18.241371 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:18.241315    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w" podUID="faa7e534-c66d-4670-83cd-c170bcc355d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 21:54:18.465430 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:18.459529    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"]
Oct 30 21:54:18.468699 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:18.468660    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:54:18.518200 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:18.518134    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"]
Oct 30 21:54:18.521487 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:18.521465    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd8f983a-9982-4ec1-82d6-a76c70838a8c-tls-certs\") pod \"distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch\" (UID: \"fd8f983a-9982-4ec1-82d6-a76c70838a8c\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:54:18.521611 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:18.521498    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd8f983a-9982-4ec1-82d6-a76c70838a8c-model-cache\") pod \"distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch\" (UID: \"fd8f983a-9982-4ec1-82d6-a76c70838a8c\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:54:18.521702 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:18.521685    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlp69\" (UniqueName: \"kubernetes.io/projected/fd8f983a-9982-4ec1-82d6-a76c70838a8c-kube-api-access-nlp69\") pod \"distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch\" (UID: \"fd8f983a-9982-4ec1-82d6-a76c70838a8c\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:54:18.521763 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:18.521732    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd8f983a-9982-4ec1-82d6-a76c70838a8c-dshm\") pod \"distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch\" (UID: \"fd8f983a-9982-4ec1-82d6-a76c70838a8c\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:54:18.521821 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:18.521792    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd8f983a-9982-4ec1-82d6-a76c70838a8c-tmp-dir\") pod \"distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch\" (UID: \"fd8f983a-9982-4ec1-82d6-a76c70838a8c\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:54:18.521821 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:18.521815    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd8f983a-9982-4ec1-82d6-a76c70838a8c-home\") pod \"distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch\" (UID: \"fd8f983a-9982-4ec1-82d6-a76c70838a8c\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:54:18.562667 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:18.562633    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-dda9f5\"/\"distinct-2-w0-dda9f5-kserve-self-signed-certs\""
Oct 30 21:54:18.622532 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:18.622500    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd8f983a-9982-4ec1-82d6-a76c70838a8c-dshm\") pod \"distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch\" (UID: \"fd8f983a-9982-4ec1-82d6-a76c70838a8c\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:54:18.622703 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:18.622549    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd8f983a-9982-4ec1-82d6-a76c70838a8c-tmp-dir\") pod \"distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch\" (UID: \"fd8f983a-9982-4ec1-82d6-a76c70838a8c\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:54:18.622703 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:18.622668    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd8f983a-9982-4ec1-82d6-a76c70838a8c-home\") pod \"distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch\" (UID: \"fd8f983a-9982-4ec1-82d6-a76c70838a8c\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:54:18.622775 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:18.622712    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd8f983a-9982-4ec1-82d6-a76c70838a8c-tls-certs\") pod \"distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch\" (UID: \"fd8f983a-9982-4ec1-82d6-a76c70838a8c\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:54:18.622775 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:18.622741    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd8f983a-9982-4ec1-82d6-a76c70838a8c-model-cache\") pod \"distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch\" (UID: \"fd8f983a-9982-4ec1-82d6-a76c70838a8c\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:54:18.622889 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:18.622875    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlp69\" (UniqueName: \"kubernetes.io/projected/fd8f983a-9982-4ec1-82d6-a76c70838a8c-kube-api-access-nlp69\") pod \"distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch\" (UID: \"fd8f983a-9982-4ec1-82d6-a76c70838a8c\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:54:18.622962 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:18.622942    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd8f983a-9982-4ec1-82d6-a76c70838a8c-tmp-dir\") pod \"distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch\" (UID: \"fd8f983a-9982-4ec1-82d6-a76c70838a8c\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:54:18.623132 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:18.623113    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd8f983a-9982-4ec1-82d6-a76c70838a8c-home\") pod \"distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch\" (UID: \"fd8f983a-9982-4ec1-82d6-a76c70838a8c\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:54:18.623239 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:18.623221    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd8f983a-9982-4ec1-82d6-a76c70838a8c-model-cache\") pod \"distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch\" (UID: \"fd8f983a-9982-4ec1-82d6-a76c70838a8c\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:54:18.629383 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:18.624939    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd8f983a-9982-4ec1-82d6-a76c70838a8c-dshm\") pod \"distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch\" (UID: \"fd8f983a-9982-4ec1-82d6-a76c70838a8c\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:54:18.629383 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:18.625464    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd8f983a-9982-4ec1-82d6-a76c70838a8c-tls-certs\") pod \"distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch\" (UID: \"fd8f983a-9982-4ec1-82d6-a76c70838a8c\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:54:18.683078 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:18.683043    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlp69\" (UniqueName: \"kubernetes.io/projected/fd8f983a-9982-4ec1-82d6-a76c70838a8c-kube-api-access-nlp69\") pod \"distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch\" (UID: \"fd8f983a-9982-4ec1-82d6-a76c70838a8c\") " pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:54:18.790421 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:18.789807    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:54:19.078036 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:19.077396    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53a6ea98-8qp8j"]
Oct 30 21:54:19.082726 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:19.081733    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53a6ea98-8qp8j"]
Oct 30 21:54:19.260374 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:19.240689    5087 patch_prober.go:29] interesting pod/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 21:54:19.260374 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:19.240747    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w" podUID="faa7e534-c66d-4670-83cd-c170bcc355d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 21:54:19.295421 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:19.295397    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"]
Oct 30 21:54:19.297259 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:54:19.297231    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd8f983a_9982_4ec1_82d6_a76c70838a8c.slice/crio-6517ac240fd75f2cdf017dc919c77edea5313c9fc7e237586865f7f796c9a66b WatchSource:0}: Error finding container 6517ac240fd75f2cdf017dc919c77edea5313c9fc7e237586865f7f796c9a66b: Status 404 returned error can't find the container with id 6517ac240fd75f2cdf017dc919c77edea5313c9fc7e237586865f7f796c9a66b
Oct 30 21:54:19.710117 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:19.710041    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7ed21eb-f88e-4f82-a19b-179c3026d74a" path="/var/lib/kubelet/pods/c7ed21eb-f88e-4f82-a19b-179c3026d74a/volumes"
Oct 30 21:54:19.768262 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:19.768229    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" event={"ID":"fd8f983a-9982-4ec1-82d6-a76c70838a8c","Type":"ContainerStarted","Data":"b7624c27434712232a8e2c80556fcfb51de0bb56293f6d35b78a0784cd3766ef"}
Oct 30 21:54:19.768463 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:19.768269    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" event={"ID":"fd8f983a-9982-4ec1-82d6-a76c70838a8c","Type":"ContainerStarted","Data":"6517ac240fd75f2cdf017dc919c77edea5313c9fc7e237586865f7f796c9a66b"}
Oct 30 21:54:19.789866 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:19.789840    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:54:19.888632 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:19.888576    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" podStartSLOduration=1.888561234 podStartE2EDuration="1.888561234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:19.88450787 +0000 UTC m=+1254.359011855" watchObservedRunningTime="2026-10-30 21:54:19.888561234 +0000 UTC m=+1254.363065212"
Oct 30 21:54:20.241767 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:20.241726    5087 patch_prober.go:29] interesting pod/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 21:54:20.242067 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:20.242033    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w" podUID="faa7e534-c66d-4670-83cd-c170bcc355d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 21:54:21.240767 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:21.240729    5087 patch_prober.go:29] interesting pod/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 21:54:21.241529 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:21.241497    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w" podUID="faa7e534-c66d-4670-83cd-c170bcc355d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 21:54:22.237736 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:22.237693    5087 patch_prober.go:29] interesting pod/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 30 21:54:22.237925 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:22.237775    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w" podUID="faa7e534-c66d-4670-83cd-c170bcc355d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 30 21:54:23.242577 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:23.242545    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:23.243058 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:23.242945    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:23.243674 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:23.243654    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 21:54:28.747561 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:28.747519    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"]
Oct 30 21:54:28.753135 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:28.753114    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:28.775155 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:28.775125    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-workload-socket\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:28.775298 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:28.775171    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-credential-socket\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:28.775377 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:28.775332    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5wn4\" (UniqueName: \"kubernetes.io/projected/3f00bbc1-19c4-447a-9459-3443bbabb845-kube-api-access-k5wn4\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:28.775450 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:28.775430    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-workload-certs\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:28.775525 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:28.775511    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f00bbc1-19c4-447a-9459-3443bbabb845-istio-podinfo\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:28.775578 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:28.775540    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f00bbc1-19c4-447a-9459-3443bbabb845-istio-token\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:28.775645 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:28.775620    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f00bbc1-19c4-447a-9459-3443bbabb845-istiod-ca-cert\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:28.775732 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:28.775715    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-istio-envoy\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:28.775785 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:28.775747    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-istio-data\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:28.790432 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:28.790409    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:54:28.807428 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:28.807409    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:54:28.828018 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:28.827989    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-c3e2d8-openshift-default-dockercfg-hqs2q\""
Oct 30 21:54:28.875466 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:28.875438    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:54:28.876973 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:28.876935    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-workload-socket\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:28.877100 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:28.876986    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-credential-socket\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:28.877100 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:28.877079    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5wn4\" (UniqueName: \"kubernetes.io/projected/3f00bbc1-19c4-447a-9459-3443bbabb845-kube-api-access-k5wn4\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:28.877247 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:28.877140    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-workload-certs\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:28.877247 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:28.877229    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f00bbc1-19c4-447a-9459-3443bbabb845-istio-podinfo\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:28.877397 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:28.877253    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f00bbc1-19c4-447a-9459-3443bbabb845-istio-token\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:28.877397 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:28.877282    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f00bbc1-19c4-447a-9459-3443bbabb845-istiod-ca-cert\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:28.877397 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:28.877380    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-istio-envoy\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:28.877397 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:28.877387    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-workload-socket\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:28.877625 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:28.877418    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-istio-data\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:29.003240 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:29.003057    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-credential-socket\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:29.041753 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:29.041722    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"]
Oct 30 21:54:29.053762 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:29.052689    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-istio-data\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:29.060242 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:29.056957    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-workload-certs\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:29.060242 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:29.060125    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5wn4\" (UniqueName: \"kubernetes.io/projected/3f00bbc1-19c4-447a-9459-3443bbabb845-kube-api-access-k5wn4\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:29.062822 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:29.061102    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f00bbc1-19c4-447a-9459-3443bbabb845-istio-podinfo\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:29.138286 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:29.138245    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f00bbc1-19c4-447a-9459-3443bbabb845-istiod-ca-cert\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:29.202370 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:29.202295    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-istio-envoy\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:29.204328 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:29.204200    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f00bbc1-19c4-447a-9459-3443bbabb845-istio-token\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:29.368048 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:29.367952    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:29.835512 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:54:29.835484    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f00bbc1_19c4_447a_9459_3443bbabb845.slice/crio-6f94ac56b324f9a012b9d60b9e68a81f708f12c2e193ff2eff8750900b358cf5 WatchSource:0}: Error finding container 6f94ac56b324f9a012b9d60b9e68a81f708f12c2e193ff2eff8750900b358cf5: Status 404 returned error can't find the container with id 6f94ac56b324f9a012b9d60b9e68a81f708f12c2e193ff2eff8750900b358cf5
Oct 30 21:54:29.835953 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:29.835738    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"]
Oct 30 21:54:29.853573 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:29.851204    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:54:29.853573 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:29.851288    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:54:29.853573 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:29.851367    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:54:29.939271 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:29.939234    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx" event={"ID":"3f00bbc1-19c4-447a-9459-3443bbabb845","Type":"ContainerStarted","Data":"6f94ac56b324f9a012b9d60b9e68a81f708f12c2e193ff2eff8750900b358cf5"}
Oct 30 21:54:30.966373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:30.965701    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx" event={"ID":"3f00bbc1-19c4-447a-9459-3443bbabb845","Type":"ContainerStarted","Data":"e381a93f014c13451aa6bb404250a935c9e75c9801b62a8ba9adfbd1d61a8892"}
Oct 30 21:54:31.171389 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:31.167298    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx" podStartSLOduration=3.167267884 podStartE2EDuration="3.167267884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:31.140828799 +0000 UTC m=+1265.615332840" watchObservedRunningTime="2026-10-30 21:54:31.167267884 +0000 UTC m=+1265.641771863"
Oct 30 21:54:31.371606 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:31.371569    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:31.375373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:31.375324    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 21:54:31.375999 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:31.375558    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx" podUID="3f00bbc1-19c4-447a-9459-3443bbabb845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 21:54:32.368521 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:32.368487    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 21:54:32.369017 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:32.368544    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx" podUID="3f00bbc1-19c4-447a-9459-3443bbabb845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 21:54:33.368639 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:33.368606    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 21:54:33.369136 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:33.368666    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx" podUID="3f00bbc1-19c4-447a-9459-3443bbabb845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 21:54:34.368813 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:34.368773    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 21:54:34.369328 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:34.368848    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx" podUID="3f00bbc1-19c4-447a-9459-3443bbabb845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 21:54:35.062480 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:35.062444    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5109e7be-q8vpg"]
Oct 30 21:54:35.105922 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:35.105891    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5109e7be-q8vpg"]
Oct 30 21:54:35.369434 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:35.368769    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 21:54:35.369434 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:35.368835    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx" podUID="3f00bbc1-19c4-447a-9459-3443bbabb845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 21:54:35.711532 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:35.711455    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ae089fd-bbb8-49d6-b323-0f8dac4b91d7" path="/var/lib/kubelet/pods/3ae089fd-bbb8-49d6-b323-0f8dac4b91d7/volumes"
Oct 30 21:54:36.368783 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:36.368723    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Oct 30 21:54:36.369014 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:36.368804    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx" podUID="3f00bbc1-19c4-447a-9459-3443bbabb845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Oct 30 21:54:36.627012 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:36.626919    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59cb4cce-x7z8c"]
Oct 30 21:54:36.631206 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:36.631182    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cb4cce-x7z8c"
Oct 30 21:54:36.733769 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:36.733728    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:54:36.733769 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:36.733785    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:54:36.739888 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:36.739855    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59cb4cce-x7z8c"]
Oct 30 21:54:36.796618 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:36.796574    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7682ce92-4eb7-42b2-8d7d-8b898652367f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59cb4cce-x7z8c\" (UID: \"7682ce92-4eb7-42b2-8d7d-8b898652367f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cb4cce-x7z8c"
Oct 30 21:54:36.796822 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:36.796623    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfp7r\" (UniqueName: \"kubernetes.io/projected/7682ce92-4eb7-42b2-8d7d-8b898652367f-kube-api-access-sfp7r\") pod \"maas-api-revoke-sub-59cb4cce-x7z8c\" (UID: \"7682ce92-4eb7-42b2-8d7d-8b898652367f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cb4cce-x7z8c"
Oct 30 21:54:36.796822 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:36.796744    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7682ce92-4eb7-42b2-8d7d-8b898652367f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59cb4cce-x7z8c\" (UID: \"7682ce92-4eb7-42b2-8d7d-8b898652367f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cb4cce-x7z8c"
Oct 30 21:54:36.898236 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:36.898147    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7682ce92-4eb7-42b2-8d7d-8b898652367f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59cb4cce-x7z8c\" (UID: \"7682ce92-4eb7-42b2-8d7d-8b898652367f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cb4cce-x7z8c"
Oct 30 21:54:36.898442 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:36.898336    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7682ce92-4eb7-42b2-8d7d-8b898652367f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59cb4cce-x7z8c\" (UID: \"7682ce92-4eb7-42b2-8d7d-8b898652367f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cb4cce-x7z8c"
Oct 30 21:54:36.898442 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:36.898382    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfp7r\" (UniqueName: \"kubernetes.io/projected/7682ce92-4eb7-42b2-8d7d-8b898652367f-kube-api-access-sfp7r\") pod \"maas-api-revoke-sub-59cb4cce-x7z8c\" (UID: \"7682ce92-4eb7-42b2-8d7d-8b898652367f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cb4cce-x7z8c"
Oct 30 21:54:36.898891 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:36.898867    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7682ce92-4eb7-42b2-8d7d-8b898652367f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59cb4cce-x7z8c\" (UID: \"7682ce92-4eb7-42b2-8d7d-8b898652367f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cb4cce-x7z8c"
Oct 30 21:54:36.900907 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:36.900886    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7682ce92-4eb7-42b2-8d7d-8b898652367f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59cb4cce-x7z8c\" (UID: \"7682ce92-4eb7-42b2-8d7d-8b898652367f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cb4cce-x7z8c"
Oct 30 21:54:36.955699 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:36.955667    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfp7r\" (UniqueName: \"kubernetes.io/projected/7682ce92-4eb7-42b2-8d7d-8b898652367f-kube-api-access-sfp7r\") pod \"maas-api-revoke-sub-59cb4cce-x7z8c\" (UID: \"7682ce92-4eb7-42b2-8d7d-8b898652367f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cb4cce-x7z8c"
Oct 30 21:54:37.246261 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:37.246220    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cb4cce-x7z8c"
Oct 30 21:54:37.667262 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:37.667215    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59cb4cce-x7z8c"]
Oct 30 21:54:37.761370 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:37.758084    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:37.761370 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:37.759832    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:37.765392 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:37.762019    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 21:54:38.028978 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:38.028936    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cb4cce-x7z8c" event={"ID":"7682ce92-4eb7-42b2-8d7d-8b898652367f","Type":"ContainerStarted","Data":"fa7fc80ac40cb78aa9950255992325e4df353183f857a8ae3c6408619dda0f69"}
Oct 30 21:54:38.029162 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:38.029000    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cb4cce-x7z8c" event={"ID":"7682ce92-4eb7-42b2-8d7d-8b898652367f","Type":"ContainerStarted","Data":"5d3b84ea053eb1d83a26b0262c1be018e2c7b24b3f7ede28e0e5f7ae54a3a8c9"}
Oct 30 21:54:38.143535 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:38.143462    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cb4cce-x7z8c" podStartSLOduration=2.143423849 podStartE2EDuration="2.143423849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:38.142850477 +0000 UTC m=+1272.617354457" watchObservedRunningTime="2026-10-30 21:54:38.143423849 +0000 UTC m=+1272.617927829"
Oct 30 21:54:39.036700 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:39.036669    5087 generic.go:356] "Generic (PLEG): container finished" podID="7682ce92-4eb7-42b2-8d7d-8b898652367f" containerID="fa7fc80ac40cb78aa9950255992325e4df353183f857a8ae3c6408619dda0f69" exitCode=0
Oct 30 21:54:39.037124 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:39.036758    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cb4cce-x7z8c" event={"ID":"7682ce92-4eb7-42b2-8d7d-8b898652367f","Type":"ContainerDied","Data":"fa7fc80ac40cb78aa9950255992325e4df353183f857a8ae3c6408619dda0f69"}
Oct 30 21:54:40.208668 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:40.208638    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cb4cce-x7z8c"
Oct 30 21:54:40.281991 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:40.281961    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7682ce92-4eb7-42b2-8d7d-8b898652367f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7682ce92-4eb7-42b2-8d7d-8b898652367f-maas-api-openshift-ca\") pod \"7682ce92-4eb7-42b2-8d7d-8b898652367f\" (UID: \"7682ce92-4eb7-42b2-8d7d-8b898652367f\") "
Oct 30 21:54:40.282154 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:40.282002    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7682ce92-4eb7-42b2-8d7d-8b898652367f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7682ce92-4eb7-42b2-8d7d-8b898652367f-maas-api-xks-ca\") pod \"7682ce92-4eb7-42b2-8d7d-8b898652367f\" (UID: \"7682ce92-4eb7-42b2-8d7d-8b898652367f\") "
Oct 30 21:54:40.282154 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:40.282069    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7682ce92-4eb7-42b2-8d7d-8b898652367f-kube-api-access-sfp7r\" (UniqueName: \"kubernetes.io/projected/7682ce92-4eb7-42b2-8d7d-8b898652367f-kube-api-access-sfp7r\") pod \"7682ce92-4eb7-42b2-8d7d-8b898652367f\" (UID: \"7682ce92-4eb7-42b2-8d7d-8b898652367f\") "
Oct 30 21:54:40.282413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:40.282390    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7682ce92-4eb7-42b2-8d7d-8b898652367f-maas-api-openshift-ca" pod "7682ce92-4eb7-42b2-8d7d-8b898652367f" (UID: "7682ce92-4eb7-42b2-8d7d-8b898652367f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:40.282606 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:40.282591    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7682ce92-4eb7-42b2-8d7d-8b898652367f-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:40.284280 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:40.284253    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7682ce92-4eb7-42b2-8d7d-8b898652367f-maas-api-xks-ca" pod "7682ce92-4eb7-42b2-8d7d-8b898652367f" (UID: "7682ce92-4eb7-42b2-8d7d-8b898652367f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:40.284280 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:40.284264    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7682ce92-4eb7-42b2-8d7d-8b898652367f-kube-api-access-sfp7r" pod "7682ce92-4eb7-42b2-8d7d-8b898652367f" (UID: "7682ce92-4eb7-42b2-8d7d-8b898652367f"). InnerVolumeSpecName "kube-api-access-sfp7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:40.383797 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:40.383730    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfp7r\" (UniqueName: \"kubernetes.io/projected/7682ce92-4eb7-42b2-8d7d-8b898652367f-kube-api-access-sfp7r\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:40.383797 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:40.383756    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7682ce92-4eb7-42b2-8d7d-8b898652367f-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:41.051227 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:41.051195    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cb4cce-x7z8c" event={"ID":"7682ce92-4eb7-42b2-8d7d-8b898652367f","Type":"ContainerDied","Data":"5d3b84ea053eb1d83a26b0262c1be018e2c7b24b3f7ede28e0e5f7ae54a3a8c9"}
Oct 30 21:54:41.051227 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:41.051217    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cb4cce-x7z8c"
Oct 30 21:54:41.051479 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:41.051269    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d3b84ea053eb1d83a26b0262c1be018e2c7b24b3f7ede28e0e5f7ae54a3a8c9"
Oct 30 21:54:45.239192 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.239159    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"]
Oct 30 21:54:45.239754 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.239737    5087 cpu_manager.go:414] "containerMap: removing container" podUID="7682ce92-4eb7-42b2-8d7d-8b898652367f" containerName="revoke-keys"
Oct 30 21:54:45.239754 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.239754    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7682ce92-4eb7-42b2-8d7d-8b898652367f" containerName="revoke-keys"
Oct 30 21:54:45.239860 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.239851    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="7682ce92-4eb7-42b2-8d7d-8b898652367f" containerName="revoke-keys"
Oct 30 21:54:45.243585 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.243565    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:45.341827 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.341798    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"]
Oct 30 21:54:45.343084 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.343050    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-istio-envoy\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:45.343223 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.343112    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-workload-socket\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:45.343223 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.343200    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-workload-certs\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:45.343385 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.343230    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/322d51a7-324e-4f33-a606-3384bb72d3cc-istio-podinfo\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:45.343385 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.343259    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/322d51a7-324e-4f33-a606-3384bb72d3cc-istiod-ca-cert\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:45.343385 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.343314    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-istio-data\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:45.343579 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.343383    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/322d51a7-324e-4f33-a606-3384bb72d3cc-istio-token\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:45.343579 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.343489    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-credential-socket\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:45.343579 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.343513    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwwcr\" (UniqueName: \"kubernetes.io/projected/322d51a7-324e-4f33-a606-3384bb72d3cc-kube-api-access-xwwcr\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:45.445019 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.444982    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-workload-socket\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:45.445188 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.445025    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-workload-certs\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:45.445188 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.445065    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/322d51a7-324e-4f33-a606-3384bb72d3cc-istio-podinfo\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:45.445188 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.445107    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/322d51a7-324e-4f33-a606-3384bb72d3cc-istiod-ca-cert\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:45.445188 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.445173    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-istio-data\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:45.445419 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.445228    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/322d51a7-324e-4f33-a606-3384bb72d3cc-istio-token\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:45.445419 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.445332    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-credential-socket\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:45.445419 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.445375    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwwcr\" (UniqueName: \"kubernetes.io/projected/322d51a7-324e-4f33-a606-3384bb72d3cc-kube-api-access-xwwcr\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:45.445586 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.445440    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-istio-envoy\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:45.445586 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.445447    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-workload-socket\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:45.445699 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.445644    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-istio-data\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:45.445742 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.445694    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-credential-socket\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:45.445835 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.445807    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-workload-certs\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:45.446208 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.446177    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/322d51a7-324e-4f33-a606-3384bb72d3cc-istiod-ca-cert\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:45.447814 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.447788    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-istio-envoy\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:45.448239 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.448220    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/322d51a7-324e-4f33-a606-3384bb72d3cc-istio-podinfo\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:45.502782 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.502707    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/322d51a7-324e-4f33-a606-3384bb72d3cc-istio-token\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:45.503010 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.502989    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwwcr\" (UniqueName: \"kubernetes.io/projected/322d51a7-324e-4f33-a606-3384bb72d3cc-kube-api-access-xwwcr\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:45.557738 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.557708    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:45.992066 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:54:45.992037    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322d51a7_324e_4f33_a606_3384bb72d3cc.slice/crio-826b9963729dfbbc5b983f507b5073017cab3c63ea57ffdbae54b3fa27fe1686 WatchSource:0}: Error finding container 826b9963729dfbbc5b983f507b5073017cab3c63ea57ffdbae54b3fa27fe1686: Status 404 returned error can't find the container with id 826b9963729dfbbc5b983f507b5073017cab3c63ea57ffdbae54b3fa27fe1686
Oct 30 21:54:45.992943 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.992915    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"]
Oct 30 21:54:45.994833 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.994800    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:54:45.994911 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.994876    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:54:45.994956 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:45.994921    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:54:46.089079 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:46.089043    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc" event={"ID":"322d51a7-324e-4f33-a606-3384bb72d3cc","Type":"ContainerStarted","Data":"083569d08e187f3975f006cc2162191cc791ca68796421440dd8ff3e28df75cf"}
Oct 30 21:54:46.089231 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:46.089106    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc" event={"ID":"322d51a7-324e-4f33-a606-3384bb72d3cc","Type":"ContainerStarted","Data":"826b9963729dfbbc5b983f507b5073017cab3c63ea57ffdbae54b3fa27fe1686"}
Oct 30 21:54:46.213181 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:46.213130    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc" podStartSLOduration=1.213116414 podStartE2EDuration="1.213116414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:46.211481442 +0000 UTC m=+1280.685985421" watchObservedRunningTime="2026-10-30 21:54:46.213116414 +0000 UTC m=+1280.687620392"
Oct 30 21:54:46.561538 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:46.558864    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:47.204421 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:54:47.204380    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:54:47.423498 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:47.423462    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a37b75cf-rbnsg"]
Oct 30 21:54:47.427611 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:47.427586    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b75cf-rbnsg"
Oct 30 21:54:47.484432 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:47.480173    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d397505-2b12-4c67-8c4e-3e202bd77442-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a37b75cf-rbnsg\" (UID: \"3d397505-2b12-4c67-8c4e-3e202bd77442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b75cf-rbnsg"
Oct 30 21:54:47.484432 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:47.480375    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8659b\" (UniqueName: \"kubernetes.io/projected/3d397505-2b12-4c67-8c4e-3e202bd77442-kube-api-access-8659b\") pod \"maas-api-revoke-sub-a37b75cf-rbnsg\" (UID: \"3d397505-2b12-4c67-8c4e-3e202bd77442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b75cf-rbnsg"
Oct 30 21:54:47.484432 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:47.480423    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d397505-2b12-4c67-8c4e-3e202bd77442-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a37b75cf-rbnsg\" (UID: \"3d397505-2b12-4c67-8c4e-3e202bd77442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b75cf-rbnsg"
Oct 30 21:54:47.487817 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:47.487755    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a37b75cf-rbnsg"]
Oct 30 21:54:47.534387 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:47.533312    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:54:47.534387 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:47.533338    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:54:47.569810 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:47.569446    5087 patch_prober.go:29] interesting pod/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 21:54:47.569810 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:47.569530    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc" podUID="322d51a7-324e-4f33-a606-3384bb72d3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 21:54:47.623564 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:47.583211    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8659b\" (UniqueName: \"kubernetes.io/projected/3d397505-2b12-4c67-8c4e-3e202bd77442-kube-api-access-8659b\") pod \"maas-api-revoke-sub-a37b75cf-rbnsg\" (UID: \"3d397505-2b12-4c67-8c4e-3e202bd77442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b75cf-rbnsg"
Oct 30 21:54:47.623564 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:47.583283    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d397505-2b12-4c67-8c4e-3e202bd77442-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a37b75cf-rbnsg\" (UID: \"3d397505-2b12-4c67-8c4e-3e202bd77442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b75cf-rbnsg"
Oct 30 21:54:47.623564 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:47.583448    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d397505-2b12-4c67-8c4e-3e202bd77442-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a37b75cf-rbnsg\" (UID: \"3d397505-2b12-4c67-8c4e-3e202bd77442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b75cf-rbnsg"
Oct 30 21:54:47.624301 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:47.624272    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d397505-2b12-4c67-8c4e-3e202bd77442-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a37b75cf-rbnsg\" (UID: \"3d397505-2b12-4c67-8c4e-3e202bd77442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b75cf-rbnsg"
Oct 30 21:54:47.634808 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:47.634773    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d397505-2b12-4c67-8c4e-3e202bd77442-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a37b75cf-rbnsg\" (UID: \"3d397505-2b12-4c67-8c4e-3e202bd77442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b75cf-rbnsg"
Oct 30 21:54:47.715623 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:47.714146    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8659b\" (UniqueName: \"kubernetes.io/projected/3d397505-2b12-4c67-8c4e-3e202bd77442-kube-api-access-8659b\") pod \"maas-api-revoke-sub-a37b75cf-rbnsg\" (UID: \"3d397505-2b12-4c67-8c4e-3e202bd77442\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b75cf-rbnsg"
Oct 30 21:54:47.772427 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:47.771545    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b75cf-rbnsg"
Oct 30 21:54:48.209700 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:48.209614    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a37b75cf-rbnsg"]
Oct 30 21:54:48.222013 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:54:48.221982    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d397505_2b12_4c67_8c4e_3e202bd77442.slice/crio-4ecf67bbe40229b9cf3613bfae8ae54e4187db8d3e0146a143588e96ee964508 WatchSource:0}: Error finding container 4ecf67bbe40229b9cf3613bfae8ae54e4187db8d3e0146a143588e96ee964508: Status 404 returned error can't find the container with id 4ecf67bbe40229b9cf3613bfae8ae54e4187db8d3e0146a143588e96ee964508
Oct 30 21:54:48.572532 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:48.572493    5087 patch_prober.go:29] interesting pod/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 21:54:48.573126 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:48.573093    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc" podUID="322d51a7-324e-4f33-a606-3384bb72d3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 21:54:49.112540 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:49.112507    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b75cf-rbnsg" event={"ID":"3d397505-2b12-4c67-8c4e-3e202bd77442","Type":"ContainerStarted","Data":"1e38ecf796ea47df31caefc0278dd0e0346346057873765ce0a854da11a0afa8"}
Oct 30 21:54:49.112718 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:49.112554    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b75cf-rbnsg" event={"ID":"3d397505-2b12-4c67-8c4e-3e202bd77442","Type":"ContainerStarted","Data":"4ecf67bbe40229b9cf3613bfae8ae54e4187db8d3e0146a143588e96ee964508"}
Oct 30 21:54:49.230264 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:49.230206    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b75cf-rbnsg" podStartSLOduration=2.230191624 podStartE2EDuration="2.230191624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:49.228607344 +0000 UTC m=+1283.703111327" watchObservedRunningTime="2026-10-30 21:54:49.230191624 +0000 UTC m=+1283.704695603"
Oct 30 21:54:49.561678 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:49.561633    5087 patch_prober.go:29] interesting pod/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 21:54:49.561874 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:49.561691    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc" podUID="322d51a7-324e-4f33-a606-3384bb72d3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 21:54:50.121280 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:50.121239    5087 generic.go:356] "Generic (PLEG): container finished" podID="3d397505-2b12-4c67-8c4e-3e202bd77442" containerID="1e38ecf796ea47df31caefc0278dd0e0346346057873765ce0a854da11a0afa8" exitCode=0
Oct 30 21:54:50.121691 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:50.121327    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b75cf-rbnsg" event={"ID":"3d397505-2b12-4c67-8c4e-3e202bd77442","Type":"ContainerDied","Data":"1e38ecf796ea47df31caefc0278dd0e0346346057873765ce0a854da11a0afa8"}
Oct 30 21:54:50.560375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:50.559639    5087 patch_prober.go:29] interesting pod/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 21:54:50.560375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:50.559699    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc" podUID="322d51a7-324e-4f33-a606-3384bb72d3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 21:54:51.345367 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:51.336479    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b75cf-rbnsg"
Oct 30 21:54:51.408161 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:51.408070    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d397505-2b12-4c67-8c4e-3e202bd77442-kube-api-access-8659b\" (UniqueName: \"kubernetes.io/projected/3d397505-2b12-4c67-8c4e-3e202bd77442-kube-api-access-8659b\") pod \"3d397505-2b12-4c67-8c4e-3e202bd77442\" (UID: \"3d397505-2b12-4c67-8c4e-3e202bd77442\") "
Oct 30 21:54:51.408341 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:51.408199    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d397505-2b12-4c67-8c4e-3e202bd77442-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d397505-2b12-4c67-8c4e-3e202bd77442-maas-api-openshift-ca\") pod \"3d397505-2b12-4c67-8c4e-3e202bd77442\" (UID: \"3d397505-2b12-4c67-8c4e-3e202bd77442\") "
Oct 30 21:54:51.408341 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:51.408281    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d397505-2b12-4c67-8c4e-3e202bd77442-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d397505-2b12-4c67-8c4e-3e202bd77442-maas-api-xks-ca\") pod \"3d397505-2b12-4c67-8c4e-3e202bd77442\" (UID: \"3d397505-2b12-4c67-8c4e-3e202bd77442\") "
Oct 30 21:54:51.409134 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:51.409109    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d397505-2b12-4c67-8c4e-3e202bd77442-maas-api-openshift-ca" pod "3d397505-2b12-4c67-8c4e-3e202bd77442" (UID: "3d397505-2b12-4c67-8c4e-3e202bd77442"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:51.413378 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:51.413314    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d397505-2b12-4c67-8c4e-3e202bd77442-maas-api-xks-ca" pod "3d397505-2b12-4c67-8c4e-3e202bd77442" (UID: "3d397505-2b12-4c67-8c4e-3e202bd77442"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:51.413529 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:51.413320    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d397505-2b12-4c67-8c4e-3e202bd77442-kube-api-access-8659b" pod "3d397505-2b12-4c67-8c4e-3e202bd77442" (UID: "3d397505-2b12-4c67-8c4e-3e202bd77442"). InnerVolumeSpecName "kube-api-access-8659b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:51.509737 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:51.509707    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d397505-2b12-4c67-8c4e-3e202bd77442-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:51.509980 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:51.509959    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d397505-2b12-4c67-8c4e-3e202bd77442-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:51.510101 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:51.510086    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8659b\" (UniqueName: \"kubernetes.io/projected/3d397505-2b12-4c67-8c4e-3e202bd77442-kube-api-access-8659b\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:54:51.558695 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:51.558205    5087 patch_prober.go:29] interesting pod/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 21:54:51.558695 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:51.558265    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc" podUID="322d51a7-324e-4f33-a606-3384bb72d3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 21:54:52.138122 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:52.138091    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b75cf-rbnsg"
Oct 30 21:54:52.138122 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:52.138106    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a37b75cf-rbnsg" event={"ID":"3d397505-2b12-4c67-8c4e-3e202bd77442","Type":"ContainerDied","Data":"4ecf67bbe40229b9cf3613bfae8ae54e4187db8d3e0146a143588e96ee964508"}
Oct 30 21:54:52.138335 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:52.138173    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ecf67bbe40229b9cf3613bfae8ae54e4187db8d3e0146a143588e96ee964508"
Oct 30 21:54:52.558450 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:52.558417    5087 patch_prober.go:29] interesting pod/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 30 21:54:52.558967 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:52.558479    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc" podUID="322d51a7-324e-4f33-a606-3384bb72d3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 30 21:54:53.562441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:53.562403    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:53.562926 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:53.562909    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:54:53.563725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:54:53.563702    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 21:55:00.254810 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.254775    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"]
Oct 30 21:55:00.255376 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.255340    5087 cpu_manager.go:414] "containerMap: removing container" podUID="3d397505-2b12-4c67-8c4e-3e202bd77442" containerName="revoke-keys"
Oct 30 21:55:00.255376 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.255371    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d397505-2b12-4c67-8c4e-3e202bd77442" containerName="revoke-keys"
Oct 30 21:55:00.255518 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.255508    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d397505-2b12-4c67-8c4e-3e202bd77442" containerName="revoke-keys"
Oct 30 21:55:00.259177 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.259154    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:00.331503 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.331467    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-workload-socket\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:00.331704 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.331526    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-credential-socket\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:00.331704 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.331666    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed464f0-0b79-467a-b131-fa6944d01941-istio-podinfo\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:00.331859 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.331699    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aed464f0-0b79-467a-b131-fa6944d01941-istio-token\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:00.331859 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.331836    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-istio-envoy\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:00.331993 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.331876    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvncm\" (UniqueName: \"kubernetes.io/projected/aed464f0-0b79-467a-b131-fa6944d01941-kube-api-access-rvncm\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:00.331993 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.331926    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-workload-certs\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:00.331993 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.331968    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-istio-data\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:00.332146 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.332016    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed464f0-0b79-467a-b131-fa6944d01941-istiod-ca-cert\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:00.437843 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.437802    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed464f0-0b79-467a-b131-fa6944d01941-istio-podinfo\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:00.484368 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.482214    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aed464f0-0b79-467a-b131-fa6944d01941-istio-token\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:00.484368 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.482309    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-istio-envoy\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:00.484368 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.482336    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvncm\" (UniqueName: \"kubernetes.io/projected/aed464f0-0b79-467a-b131-fa6944d01941-kube-api-access-rvncm\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:00.484368 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.482414    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-workload-certs\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:00.484368 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.482465    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-istio-data\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:00.484368 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.482517    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed464f0-0b79-467a-b131-fa6944d01941-istiod-ca-cert\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:00.484368 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.482658    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-workload-socket\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:00.484368 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.482708    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-credential-socket\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:00.484368 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.483202    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-workload-certs\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:00.493564 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.492038    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed464f0-0b79-467a-b131-fa6944d01941-istiod-ca-cert\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:00.493564 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.492335    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-istio-data\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:00.498209 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.493839    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-workload-socket\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:00.537001 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.536130    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"]
Oct 30 21:55:00.537001 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.536683    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-istio-envoy\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:00.544393 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.542915    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-credential-socket\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:00.586103 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.586030    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aed464f0-0b79-467a-b131-fa6944d01941-istio-token\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:00.587125 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.587068    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed464f0-0b79-467a-b131-fa6944d01941-istio-podinfo\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:00.590223 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.590163    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvncm\" (UniqueName: \"kubernetes.io/projected/aed464f0-0b79-467a-b131-fa6944d01941-kube-api-access-rvncm\") pod \"e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") " pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:00.875154 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:00.875064    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:01.938424 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:01.937933    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"]
Oct 30 21:55:02.045057 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:02.043120    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:55:02.045057 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:02.043221    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:55:02.045057 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:02.043266    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:55:02.314243 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:02.314203    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp" event={"ID":"aed464f0-0b79-467a-b131-fa6944d01941","Type":"ContainerStarted","Data":"b73c2ca4ec3fadbd01d458b2a8e949a8e59cee8000bc4fa5277fccd20e677f2d"}
Oct 30 21:55:02.464380 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:02.459431    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:02.488397 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:02.477442    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:02.545374 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:02.541428    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:02.628412 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:02.626508    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:02.751276 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:02.750460    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:02.799137 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:02.799084    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:02.889487 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:02.889314    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:02.895119 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:02.894414    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:02.942713 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:02.930497    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:02.968594 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:02.967735    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:55:03.074588 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:03.074443    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc" podUID="f6a9f73b-bb1f-4270-a38d-3703c46fae07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:03.128376 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:03.122415    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:03.471426 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:03.470940    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:03.486007 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:03.471862    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:55:03.486007 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:03.479411    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:03.486007 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:03.479766    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:55:03.541494 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:03.505062    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp" event={"ID":"aed464f0-0b79-467a-b131-fa6944d01941","Type":"ContainerStarted","Data":"a82229b4acff245f57189518bbc0d179e406e82a54c2debb702dff2a9233a7e5"}
Oct 30 21:55:03.666719 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:03.666663    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Oct 30 21:55:03.672633 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:03.672602    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:55:03.839401 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:03.754507    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:03.865366 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:03.806735    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:55:03.886135 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:03.883124    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp" podStartSLOduration=3.883103025 podStartE2EDuration="3.883103025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:03.871844836 +0000 UTC m=+1298.346348817" watchObservedRunningTime="2026-10-30 21:55:03.883103025 +0000 UTC m=+1298.357607008"
Oct 30 21:55:03.924417 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:03.921837    5087 patch_prober.go:29] interesting pod/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:55:03.924417 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:03.921895    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp" podUID="aed464f0-0b79-467a-b131-fa6944d01941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 21:55:04.008621 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:04.008589    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:04.017823 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:04.009297    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:55:04.026952 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:04.026918    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:55:04.027220 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:04.027203    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:55:04.027423 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:04.027408    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:55:04.027579 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:04.027566    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:55:04.027719 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:04.027706    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:55:04.027884 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:04.027871    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:55:04.876762 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:04.876662    5087 patch_prober.go:29] interesting pod/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:55:04.876762 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:04.876729    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp" podUID="aed464f0-0b79-467a-b131-fa6944d01941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 21:55:05.878028 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:05.877988    5087 patch_prober.go:29] interesting pod/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:55:05.879107 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:05.879074    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp" podUID="aed464f0-0b79-467a-b131-fa6944d01941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 21:55:06.876248 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:06.876211    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"]
Oct 30 21:55:06.876654 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:06.876620    5087 patch_prober.go:29] interesting pod/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:55:06.876779 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:06.876698    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp" podUID="aed464f0-0b79-467a-b131-fa6944d01941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 21:55:06.881712 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:06.881687    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:55:06.935969 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:06.935933    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7tnc\" (UniqueName: \"kubernetes.io/projected/01fdf382-6833-463a-bed9-c7c096166819-kube-api-access-j7tnc\") pod \"distinct-2-w1-80c366-kserve-58f958b885-rvsml\" (UID: \"01fdf382-6833-463a-bed9-c7c096166819\") " pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:55:06.936141 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:06.936097    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01fdf382-6833-463a-bed9-c7c096166819-dshm\") pod \"distinct-2-w1-80c366-kserve-58f958b885-rvsml\" (UID: \"01fdf382-6833-463a-bed9-c7c096166819\") " pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:55:06.936216 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:06.936150    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01fdf382-6833-463a-bed9-c7c096166819-tmp-dir\") pod \"distinct-2-w1-80c366-kserve-58f958b885-rvsml\" (UID: \"01fdf382-6833-463a-bed9-c7c096166819\") " pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:55:06.936269 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:06.936218    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01fdf382-6833-463a-bed9-c7c096166819-home\") pod \"distinct-2-w1-80c366-kserve-58f958b885-rvsml\" (UID: \"01fdf382-6833-463a-bed9-c7c096166819\") " pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:55:06.936269 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:06.936256    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01fdf382-6833-463a-bed9-c7c096166819-model-cache\") pod \"distinct-2-w1-80c366-kserve-58f958b885-rvsml\" (UID: \"01fdf382-6833-463a-bed9-c7c096166819\") " pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:55:06.947484 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:06.947403    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01fdf382-6833-463a-bed9-c7c096166819-tls-certs\") pod \"distinct-2-w1-80c366-kserve-58f958b885-rvsml\" (UID: \"01fdf382-6833-463a-bed9-c7c096166819\") " pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:55:06.971370 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:06.966604    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"]
Oct 30 21:55:06.980929 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:06.980907    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-80c366\"/\"distinct-2-w1-80c366-kserve-self-signed-certs\""
Oct 30 21:55:07.092928 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:07.087044    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7tnc\" (UniqueName: \"kubernetes.io/projected/01fdf382-6833-463a-bed9-c7c096166819-kube-api-access-j7tnc\") pod \"distinct-2-w1-80c366-kserve-58f958b885-rvsml\" (UID: \"01fdf382-6833-463a-bed9-c7c096166819\") " pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:55:07.092928 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:07.087159    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01fdf382-6833-463a-bed9-c7c096166819-dshm\") pod \"distinct-2-w1-80c366-kserve-58f958b885-rvsml\" (UID: \"01fdf382-6833-463a-bed9-c7c096166819\") " pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:55:07.092928 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:07.087186    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01fdf382-6833-463a-bed9-c7c096166819-tmp-dir\") pod \"distinct-2-w1-80c366-kserve-58f958b885-rvsml\" (UID: \"01fdf382-6833-463a-bed9-c7c096166819\") " pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:55:07.092928 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:07.087218    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01fdf382-6833-463a-bed9-c7c096166819-home\") pod \"distinct-2-w1-80c366-kserve-58f958b885-rvsml\" (UID: \"01fdf382-6833-463a-bed9-c7c096166819\") " pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:55:07.092928 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:07.087257    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01fdf382-6833-463a-bed9-c7c096166819-model-cache\") pod \"distinct-2-w1-80c366-kserve-58f958b885-rvsml\" (UID: \"01fdf382-6833-463a-bed9-c7c096166819\") " pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:55:07.092928 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:07.087390    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01fdf382-6833-463a-bed9-c7c096166819-tls-certs\") pod \"distinct-2-w1-80c366-kserve-58f958b885-rvsml\" (UID: \"01fdf382-6833-463a-bed9-c7c096166819\") " pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:55:07.095154 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:07.095121    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01fdf382-6833-463a-bed9-c7c096166819-tmp-dir\") pod \"distinct-2-w1-80c366-kserve-58f958b885-rvsml\" (UID: \"01fdf382-6833-463a-bed9-c7c096166819\") " pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:55:07.098867 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:07.098841    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01fdf382-6833-463a-bed9-c7c096166819-home\") pod \"distinct-2-w1-80c366-kserve-58f958b885-rvsml\" (UID: \"01fdf382-6833-463a-bed9-c7c096166819\") " pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:55:07.099778 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:07.099752    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01fdf382-6833-463a-bed9-c7c096166819-tls-certs\") pod \"distinct-2-w1-80c366-kserve-58f958b885-rvsml\" (UID: \"01fdf382-6833-463a-bed9-c7c096166819\") " pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:55:07.100334 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:07.100306    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01fdf382-6833-463a-bed9-c7c096166819-model-cache\") pod \"distinct-2-w1-80c366-kserve-58f958b885-rvsml\" (UID: \"01fdf382-6833-463a-bed9-c7c096166819\") " pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:55:07.101396 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:07.101378    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01fdf382-6833-463a-bed9-c7c096166819-dshm\") pod \"distinct-2-w1-80c366-kserve-58f958b885-rvsml\" (UID: \"01fdf382-6833-463a-bed9-c7c096166819\") " pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:55:07.108034 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:07.107988    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72adc758-rq4xc"]
Oct 30 21:55:07.115170 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:07.115130    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72adc758-rq4xc"]
Oct 30 21:55:07.172433 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:07.172332    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7tnc\" (UniqueName: \"kubernetes.io/projected/01fdf382-6833-463a-bed9-c7c096166819-kube-api-access-j7tnc\") pod \"distinct-2-w1-80c366-kserve-58f958b885-rvsml\" (UID: \"01fdf382-6833-463a-bed9-c7c096166819\") " pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:55:07.196896 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:07.196858    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:55:07.711124 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:07.711051    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ed3f6ca-e632-4096-ae8b-11420eaaf37a" path="/var/lib/kubelet/pods/2ed3f6ca-e632-4096-ae8b-11420eaaf37a/volumes"
Oct 30 21:55:07.770126 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:55:07.769018    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01fdf382_6833_463a_bed9_c7c096166819.slice/crio-1b91b9ac68a4641e1ee64933c30c3dfa8fe9b34ef472502ff2045329cc4f50a0 WatchSource:0}: Error finding container 1b91b9ac68a4641e1ee64933c30c3dfa8fe9b34ef472502ff2045329cc4f50a0: Status 404 returned error can't find the container with id 1b91b9ac68a4641e1ee64933c30c3dfa8fe9b34ef472502ff2045329cc4f50a0
Oct 30 21:55:07.772392 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:07.771689    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"]
Oct 30 21:55:07.877173 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:07.876960    5087 patch_prober.go:29] interesting pod/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:55:07.877173 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:07.877043    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp" podUID="aed464f0-0b79-467a-b131-fa6944d01941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 21:55:08.595650 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:08.595616    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml" event={"ID":"01fdf382-6833-463a-bed9-c7c096166819","Type":"ContainerStarted","Data":"da74c85e4036800e1d6599fb25537dfe99fff96d90c7e5a51504086cb975fb57"}
Oct 30 21:55:08.596049 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:08.595659    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml" event={"ID":"01fdf382-6833-463a-bed9-c7c096166819","Type":"ContainerStarted","Data":"1b91b9ac68a4641e1ee64933c30c3dfa8fe9b34ef472502ff2045329cc4f50a0"}
Oct 30 21:55:08.717206 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:08.717146    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml" podStartSLOduration=2.717131782 podStartE2EDuration="2.717131782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:08.715611145 +0000 UTC m=+1303.190115123" watchObservedRunningTime="2026-10-30 21:55:08.717131782 +0000 UTC m=+1303.191635814"
Oct 30 21:55:08.880462 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:08.876240    5087 patch_prober.go:29] interesting pod/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:55:08.880462 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:08.876302    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp" podUID="aed464f0-0b79-467a-b131-fa6944d01941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 21:55:09.198248 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:09.198171    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:55:09.877054 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:09.877022    5087 patch_prober.go:29] interesting pod/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 21:55:09.877456 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:09.877083    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp" podUID="aed464f0-0b79-467a-b131-fa6944d01941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 21:55:09.939604 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:09.939571    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84075163-469ss"]
Oct 30 21:55:09.944800 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:09.944781    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84075163-469ss"
Oct 30 21:55:09.998189 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:09.998158    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84075163-469ss"]
Oct 30 21:55:10.034839 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:10.034807    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc41ae4-0b46-4e51-83ec-c9224f91072d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84075163-469ss\" (UID: \"1bc41ae4-0b46-4e51-83ec-c9224f91072d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84075163-469ss"
Oct 30 21:55:10.035126 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:10.035102    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb4jf\" (UniqueName: \"kubernetes.io/projected/1bc41ae4-0b46-4e51-83ec-c9224f91072d-kube-api-access-rb4jf\") pod \"maas-api-revoke-sub-84075163-469ss\" (UID: \"1bc41ae4-0b46-4e51-83ec-c9224f91072d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84075163-469ss"
Oct 30 21:55:10.035361 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:10.034881    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:55:10.035521 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:10.035440    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc41ae4-0b46-4e51-83ec-c9224f91072d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84075163-469ss\" (UID: \"1bc41ae4-0b46-4e51-83ec-c9224f91072d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84075163-469ss"
Oct 30 21:55:10.042419 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:10.042397    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:55:10.136732 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:10.136645    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc41ae4-0b46-4e51-83ec-c9224f91072d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84075163-469ss\" (UID: \"1bc41ae4-0b46-4e51-83ec-c9224f91072d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84075163-469ss"
Oct 30 21:55:10.136868 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:10.136823    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc41ae4-0b46-4e51-83ec-c9224f91072d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84075163-469ss\" (UID: \"1bc41ae4-0b46-4e51-83ec-c9224f91072d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84075163-469ss"
Oct 30 21:55:10.136868 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:10.136853    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb4jf\" (UniqueName: \"kubernetes.io/projected/1bc41ae4-0b46-4e51-83ec-c9224f91072d-kube-api-access-rb4jf\") pod \"maas-api-revoke-sub-84075163-469ss\" (UID: \"1bc41ae4-0b46-4e51-83ec-c9224f91072d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84075163-469ss"
Oct 30 21:55:10.137483 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:10.137449    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc41ae4-0b46-4e51-83ec-c9224f91072d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84075163-469ss\" (UID: \"1bc41ae4-0b46-4e51-83ec-c9224f91072d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84075163-469ss"
Oct 30 21:55:10.139427 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:10.139402    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc41ae4-0b46-4e51-83ec-c9224f91072d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84075163-469ss\" (UID: \"1bc41ae4-0b46-4e51-83ec-c9224f91072d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84075163-469ss"
Oct 30 21:55:10.193962 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:10.193919    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb4jf\" (UniqueName: \"kubernetes.io/projected/1bc41ae4-0b46-4e51-83ec-c9224f91072d-kube-api-access-rb4jf\") pod \"maas-api-revoke-sub-84075163-469ss\" (UID: \"1bc41ae4-0b46-4e51-83ec-c9224f91072d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84075163-469ss"
Oct 30 21:55:10.265757 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:10.265727    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84075163-469ss"
Oct 30 21:55:10.652916 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:10.652885    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84075163-469ss"]
Oct 30 21:55:10.896136 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:10.896105    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:10.896683 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:10.896660    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:10.897428 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:10.897365    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 21:55:11.623363 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:11.623317    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84075163-469ss" event={"ID":"1bc41ae4-0b46-4e51-83ec-c9224f91072d","Type":"ContainerStarted","Data":"d777cb8593adef67036ac0eaa7bbe33c7b3c9207ed1623c8787fc04547fb1984"}
Oct 30 21:55:11.623521 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:11.623388    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84075163-469ss" event={"ID":"1bc41ae4-0b46-4e51-83ec-c9224f91072d","Type":"ContainerStarted","Data":"e64e996069ce3f1f21ab874e28c540bc734b085150926dc67971537abcbe651c"}
Oct 30 21:55:11.737633 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:11.737571    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84075163-469ss" podStartSLOduration=2.73755624 podStartE2EDuration="2.73755624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:11.735618473 +0000 UTC m=+1306.210122443" watchObservedRunningTime="2026-10-30 21:55:11.73755624 +0000 UTC m=+1306.212060219"
Oct 30 21:55:12.049158 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:12.047200    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e28ef0fa-8vr67"]
Oct 30 21:55:12.051747 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:12.051722    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e28ef0fa-8vr67"]
Oct 30 21:55:12.631691 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:12.631659    5087 generic.go:356] "Generic (PLEG): container finished" podID="1bc41ae4-0b46-4e51-83ec-c9224f91072d" containerID="d777cb8593adef67036ac0eaa7bbe33c7b3c9207ed1623c8787fc04547fb1984" exitCode=0
Oct 30 21:55:12.631859 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:12.631750    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84075163-469ss" event={"ID":"1bc41ae4-0b46-4e51-83ec-c9224f91072d","Type":"ContainerDied","Data":"d777cb8593adef67036ac0eaa7bbe33c7b3c9207ed1623c8787fc04547fb1984"}
Oct 30 21:55:13.725182 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:13.725150    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff9a88ba-cfeb-4554-8c08-384c1923b19b" path="/var/lib/kubelet/pods/ff9a88ba-cfeb-4554-8c08-384c1923b19b/volumes"
Oct 30 21:55:13.810682 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:13.810651    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84075163-469ss"
Oct 30 21:55:13.999740 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:13.999708    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bc41ae4-0b46-4e51-83ec-c9224f91072d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc41ae4-0b46-4e51-83ec-c9224f91072d-maas-api-openshift-ca\") pod \"1bc41ae4-0b46-4e51-83ec-c9224f91072d\" (UID: \"1bc41ae4-0b46-4e51-83ec-c9224f91072d\") "
Oct 30 21:55:13.999947 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:13.999778    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bc41ae4-0b46-4e51-83ec-c9224f91072d-kube-api-access-rb4jf\" (UniqueName: \"kubernetes.io/projected/1bc41ae4-0b46-4e51-83ec-c9224f91072d-kube-api-access-rb4jf\") pod \"1bc41ae4-0b46-4e51-83ec-c9224f91072d\" (UID: \"1bc41ae4-0b46-4e51-83ec-c9224f91072d\") "
Oct 30 21:55:13.999947 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:13.999843    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bc41ae4-0b46-4e51-83ec-c9224f91072d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc41ae4-0b46-4e51-83ec-c9224f91072d-maas-api-xks-ca\") pod \"1bc41ae4-0b46-4e51-83ec-c9224f91072d\" (UID: \"1bc41ae4-0b46-4e51-83ec-c9224f91072d\") "
Oct 30 21:55:14.000536 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:14.000507    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bc41ae4-0b46-4e51-83ec-c9224f91072d-maas-api-openshift-ca" pod "1bc41ae4-0b46-4e51-83ec-c9224f91072d" (UID: "1bc41ae4-0b46-4e51-83ec-c9224f91072d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:14.003120 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:14.003093    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bc41ae4-0b46-4e51-83ec-c9224f91072d-maas-api-xks-ca" pod "1bc41ae4-0b46-4e51-83ec-c9224f91072d" (UID: "1bc41ae4-0b46-4e51-83ec-c9224f91072d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:14.003120 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:14.003111    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bc41ae4-0b46-4e51-83ec-c9224f91072d-kube-api-access-rb4jf" pod "1bc41ae4-0b46-4e51-83ec-c9224f91072d" (UID: "1bc41ae4-0b46-4e51-83ec-c9224f91072d"). InnerVolumeSpecName "kube-api-access-rb4jf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:14.101705 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:14.101673    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bc41ae4-0b46-4e51-83ec-c9224f91072d-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:14.101705 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:14.101701    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb4jf\" (UniqueName: \"kubernetes.io/projected/1bc41ae4-0b46-4e51-83ec-c9224f91072d-kube-api-access-rb4jf\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:14.101908 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:14.101713    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bc41ae4-0b46-4e51-83ec-c9224f91072d-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:14.647795 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:14.647759    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84075163-469ss" event={"ID":"1bc41ae4-0b46-4e51-83ec-c9224f91072d","Type":"ContainerDied","Data":"e64e996069ce3f1f21ab874e28c540bc734b085150926dc67971537abcbe651c"}
Oct 30 21:55:14.647966 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:14.647771    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84075163-469ss"
Oct 30 21:55:14.647966 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:14.647859    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e64e996069ce3f1f21ab874e28c540bc734b085150926dc67971537abcbe651c"
Oct 30 21:55:14.649986 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:14.649964    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-497e3152-26zm9"]
Oct 30 21:55:14.650552 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:14.650539    5087 cpu_manager.go:414] "containerMap: removing container" podUID="1bc41ae4-0b46-4e51-83ec-c9224f91072d" containerName="revoke-keys"
Oct 30 21:55:14.650609 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:14.650555    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bc41ae4-0b46-4e51-83ec-c9224f91072d" containerName="revoke-keys"
Oct 30 21:55:14.650690 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:14.650680    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bc41ae4-0b46-4e51-83ec-c9224f91072d" containerName="revoke-keys"
Oct 30 21:55:14.654191 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:14.654167    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-497e3152-26zm9"
Oct 30 21:55:14.705987 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:14.705951    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:55:14.755478 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:14.755453    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:55:14.759525 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:14.759499    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-497e3152-26zm9"]
Oct 30 21:55:14.810615 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:14.810584    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smchg\" (UniqueName: \"kubernetes.io/projected/6514e520-3f58-46dd-9158-46cb05110a94-kube-api-access-smchg\") pod \"maas-api-revoke-sub-497e3152-26zm9\" (UID: \"6514e520-3f58-46dd-9158-46cb05110a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-497e3152-26zm9"
Oct 30 21:55:14.810794 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:14.810776    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6514e520-3f58-46dd-9158-46cb05110a94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-497e3152-26zm9\" (UID: \"6514e520-3f58-46dd-9158-46cb05110a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-497e3152-26zm9"
Oct 30 21:55:14.810931 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:14.810913    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6514e520-3f58-46dd-9158-46cb05110a94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-497e3152-26zm9\" (UID: \"6514e520-3f58-46dd-9158-46cb05110a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-497e3152-26zm9"
Oct 30 21:55:14.911736 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:14.911656    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6514e520-3f58-46dd-9158-46cb05110a94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-497e3152-26zm9\" (UID: \"6514e520-3f58-46dd-9158-46cb05110a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-497e3152-26zm9"
Oct 30 21:55:14.911736 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:14.911730    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smchg\" (UniqueName: \"kubernetes.io/projected/6514e520-3f58-46dd-9158-46cb05110a94-kube-api-access-smchg\") pod \"maas-api-revoke-sub-497e3152-26zm9\" (UID: \"6514e520-3f58-46dd-9158-46cb05110a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-497e3152-26zm9"
Oct 30 21:55:14.911979 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:14.911903    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6514e520-3f58-46dd-9158-46cb05110a94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-497e3152-26zm9\" (UID: \"6514e520-3f58-46dd-9158-46cb05110a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-497e3152-26zm9"
Oct 30 21:55:14.912326 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:14.912305    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6514e520-3f58-46dd-9158-46cb05110a94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-497e3152-26zm9\" (UID: \"6514e520-3f58-46dd-9158-46cb05110a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-497e3152-26zm9"
Oct 30 21:55:14.914485 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:14.914469    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6514e520-3f58-46dd-9158-46cb05110a94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-497e3152-26zm9\" (UID: \"6514e520-3f58-46dd-9158-46cb05110a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-497e3152-26zm9"
Oct 30 21:55:14.968740 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:14.968710    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smchg\" (UniqueName: \"kubernetes.io/projected/6514e520-3f58-46dd-9158-46cb05110a94-kube-api-access-smchg\") pod \"maas-api-revoke-sub-497e3152-26zm9\" (UID: \"6514e520-3f58-46dd-9158-46cb05110a94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-497e3152-26zm9"
Oct 30 21:55:14.969301 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:14.969286    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-497e3152-26zm9"
Oct 30 21:55:15.344987 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:15.344961    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-497e3152-26zm9"]
Oct 30 21:55:15.345499 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:55:15.345475    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6514e520_3f58_46dd_9158_46cb05110a94.slice/crio-98cdd19af661edacf30cfdd4e104c278d4ca70d181c6c12a8be0e51006a350b7 WatchSource:0}: Error finding container 98cdd19af661edacf30cfdd4e104c278d4ca70d181c6c12a8be0e51006a350b7: Status 404 returned error can't find the container with id 98cdd19af661edacf30cfdd4e104c278d4ca70d181c6c12a8be0e51006a350b7
Oct 30 21:55:15.660731 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:15.660701    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-497e3152-26zm9" event={"ID":"6514e520-3f58-46dd-9158-46cb05110a94","Type":"ContainerStarted","Data":"2da9ba7dbbce47ddd1616763c29917b80a64e9a0ad9f59a3c29a53be8062ec97"}
Oct 30 21:55:15.660864 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:15.660751    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-497e3152-26zm9" event={"ID":"6514e520-3f58-46dd-9158-46cb05110a94","Type":"ContainerStarted","Data":"98cdd19af661edacf30cfdd4e104c278d4ca70d181c6c12a8be0e51006a350b7"}
Oct 30 21:55:15.775718 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:15.775670    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-497e3152-26zm9" podStartSLOduration=1.775657776 podStartE2EDuration="1.775657776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:15.773925653 +0000 UTC m=+1310.248429645" watchObservedRunningTime="2026-10-30 21:55:15.775657776 +0000 UTC m=+1310.250161754"
Oct 30 21:55:16.669683 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:16.669604    5087 generic.go:356] "Generic (PLEG): container finished" podID="6514e520-3f58-46dd-9158-46cb05110a94" containerID="2da9ba7dbbce47ddd1616763c29917b80a64e9a0ad9f59a3c29a53be8062ec97" exitCode=0
Oct 30 21:55:16.669833 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:16.669692    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-497e3152-26zm9" event={"ID":"6514e520-3f58-46dd-9158-46cb05110a94","Type":"ContainerDied","Data":"2da9ba7dbbce47ddd1616763c29917b80a64e9a0ad9f59a3c29a53be8062ec97"}
Oct 30 21:55:17.172694 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:55:17.172653    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:55:17.197937 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:17.197908    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:55:17.211026 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:17.211005    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:55:17.689722 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:17.689697    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:55:17.899722 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:17.899691    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-497e3152-26zm9"
Oct 30 21:55:17.951582 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:17.951540    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6514e520-3f58-46dd-9158-46cb05110a94-kube-api-access-smchg\" (UniqueName: \"kubernetes.io/projected/6514e520-3f58-46dd-9158-46cb05110a94-kube-api-access-smchg\") pod \"6514e520-3f58-46dd-9158-46cb05110a94\" (UID: \"6514e520-3f58-46dd-9158-46cb05110a94\") "
Oct 30 21:55:17.951762 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:17.951615    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6514e520-3f58-46dd-9158-46cb05110a94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6514e520-3f58-46dd-9158-46cb05110a94-maas-api-openshift-ca\") pod \"6514e520-3f58-46dd-9158-46cb05110a94\" (UID: \"6514e520-3f58-46dd-9158-46cb05110a94\") "
Oct 30 21:55:17.951823 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:17.951787    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6514e520-3f58-46dd-9158-46cb05110a94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6514e520-3f58-46dd-9158-46cb05110a94-maas-api-xks-ca\") pod \"6514e520-3f58-46dd-9158-46cb05110a94\" (UID: \"6514e520-3f58-46dd-9158-46cb05110a94\") "
Oct 30 21:55:17.952093 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:17.952065    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6514e520-3f58-46dd-9158-46cb05110a94-maas-api-openshift-ca" pod "6514e520-3f58-46dd-9158-46cb05110a94" (UID: "6514e520-3f58-46dd-9158-46cb05110a94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:17.952411 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:17.952339    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6514e520-3f58-46dd-9158-46cb05110a94-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:17.953979 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:17.953955    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6514e520-3f58-46dd-9158-46cb05110a94-kube-api-access-smchg" pod "6514e520-3f58-46dd-9158-46cb05110a94" (UID: "6514e520-3f58-46dd-9158-46cb05110a94"). InnerVolumeSpecName "kube-api-access-smchg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:17.953979 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:17.953961    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6514e520-3f58-46dd-9158-46cb05110a94-maas-api-xks-ca" pod "6514e520-3f58-46dd-9158-46cb05110a94" (UID: "6514e520-3f58-46dd-9158-46cb05110a94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:18.053864 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:18.053776    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smchg\" (UniqueName: \"kubernetes.io/projected/6514e520-3f58-46dd-9158-46cb05110a94-kube-api-access-smchg\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:18.053864 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:18.053804    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6514e520-3f58-46dd-9158-46cb05110a94-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:18.689753 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:18.689724    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-497e3152-26zm9"
Oct 30 21:55:18.690231 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:18.689789    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-497e3152-26zm9" event={"ID":"6514e520-3f58-46dd-9158-46cb05110a94","Type":"ContainerDied","Data":"98cdd19af661edacf30cfdd4e104c278d4ca70d181c6c12a8be0e51006a350b7"}
Oct 30 21:55:18.690231 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:18.689845    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98cdd19af661edacf30cfdd4e104c278d4ca70d181c6c12a8be0e51006a350b7"
Oct 30 21:55:19.057368 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:19.057324    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da29607a-hqpxr"]
Oct 30 21:55:19.062187 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:19.062158    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da29607a-hqpxr"]
Oct 30 21:55:19.711198 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:19.711164    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc76d42b-2efe-4f83-8ebe-6b7d54bf455c" path="/var/lib/kubelet/pods/cc76d42b-2efe-4f83-8ebe-6b7d54bf455c/volumes"
Oct 30 21:55:23.776841 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:23.776814    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba05e518-z46fv"]
Oct 30 21:55:23.777491 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:23.777476    5087 cpu_manager.go:414] "containerMap: removing container" podUID="6514e520-3f58-46dd-9158-46cb05110a94" containerName="revoke-keys"
Oct 30 21:55:23.777554 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:23.777492    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6514e520-3f58-46dd-9158-46cb05110a94" containerName="revoke-keys"
Oct 30 21:55:23.777608 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:23.777595    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="6514e520-3f58-46dd-9158-46cb05110a94" containerName="revoke-keys"
Oct 30 21:55:23.781822 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:23.781796    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba05e518-z46fv"
Oct 30 21:55:23.826845 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:23.826820    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73e4cb78-cbd4-43aa-a9c7-5418323158db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba05e518-z46fv\" (UID: \"73e4cb78-cbd4-43aa-a9c7-5418323158db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba05e518-z46fv"
Oct 30 21:55:23.826985 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:23.826951    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hst9z\" (UniqueName: \"kubernetes.io/projected/73e4cb78-cbd4-43aa-a9c7-5418323158db-kube-api-access-hst9z\") pod \"maas-api-revoke-sub-ba05e518-z46fv\" (UID: \"73e4cb78-cbd4-43aa-a9c7-5418323158db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba05e518-z46fv"
Oct 30 21:55:23.827077 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:23.827047    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73e4cb78-cbd4-43aa-a9c7-5418323158db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba05e518-z46fv\" (UID: \"73e4cb78-cbd4-43aa-a9c7-5418323158db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba05e518-z46fv"
Oct 30 21:55:23.840515 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:23.840485    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba05e518-z46fv"]
Oct 30 21:55:23.884403 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:23.884337    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:55:23.884572 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:23.884456    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:55:23.928408 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:23.928370    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73e4cb78-cbd4-43aa-a9c7-5418323158db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba05e518-z46fv\" (UID: \"73e4cb78-cbd4-43aa-a9c7-5418323158db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba05e518-z46fv"
Oct 30 21:55:23.928586 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:23.928487    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hst9z\" (UniqueName: \"kubernetes.io/projected/73e4cb78-cbd4-43aa-a9c7-5418323158db-kube-api-access-hst9z\") pod \"maas-api-revoke-sub-ba05e518-z46fv\" (UID: \"73e4cb78-cbd4-43aa-a9c7-5418323158db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba05e518-z46fv"
Oct 30 21:55:23.928586 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:23.928577    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73e4cb78-cbd4-43aa-a9c7-5418323158db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba05e518-z46fv\" (UID: \"73e4cb78-cbd4-43aa-a9c7-5418323158db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba05e518-z46fv"
Oct 30 21:55:23.929287 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:23.929259    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73e4cb78-cbd4-43aa-a9c7-5418323158db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ba05e518-z46fv\" (UID: \"73e4cb78-cbd4-43aa-a9c7-5418323158db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba05e518-z46fv"
Oct 30 21:55:23.931040 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:23.931014    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73e4cb78-cbd4-43aa-a9c7-5418323158db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ba05e518-z46fv\" (UID: \"73e4cb78-cbd4-43aa-a9c7-5418323158db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba05e518-z46fv"
Oct 30 21:55:23.985917 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:23.985894    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hst9z\" (UniqueName: \"kubernetes.io/projected/73e4cb78-cbd4-43aa-a9c7-5418323158db-kube-api-access-hst9z\") pod \"maas-api-revoke-sub-ba05e518-z46fv\" (UID: \"73e4cb78-cbd4-43aa-a9c7-5418323158db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba05e518-z46fv"
Oct 30 21:55:24.049223 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:24.049141    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82332eeb-nnglk"]
Oct 30 21:55:24.052996 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:24.052974    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82332eeb-nnglk"]
Oct 30 21:55:24.097617 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:24.097591    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba05e518-z46fv"
Oct 30 21:55:24.477136 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:24.477094    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba05e518-z46fv"]
Oct 30 21:55:24.478883 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:55:24.478859    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e4cb78_cbd4_43aa_a9c7_5418323158db.slice/crio-bf0c416855528693eebd4b122a0edf045f987f8f2f834569d15e51bbf8bf91de WatchSource:0}: Error finding container bf0c416855528693eebd4b122a0edf045f987f8f2f834569d15e51bbf8bf91de: Status 404 returned error can't find the container with id bf0c416855528693eebd4b122a0edf045f987f8f2f834569d15e51bbf8bf91de
Oct 30 21:55:24.739270 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:24.739236    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba05e518-z46fv" event={"ID":"73e4cb78-cbd4-43aa-a9c7-5418323158db","Type":"ContainerStarted","Data":"bf0c416855528693eebd4b122a0edf045f987f8f2f834569d15e51bbf8bf91de"}
Oct 30 21:55:25.710177 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:25.710106    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="960bcf64-057c-4ef8-9d89-d61c54237828" path="/var/lib/kubelet/pods/960bcf64-057c-4ef8-9d89-d61c54237828/volumes"
Oct 30 21:55:25.748433 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:25.748397    5087 generic.go:356] "Generic (PLEG): container finished" podID="73e4cb78-cbd4-43aa-a9c7-5418323158db" containerID="0bbc91104567ec028fe9bf983fa04158ad75da1016bc8f5f5f1d6a25415bb964" exitCode=0
Oct 30 21:55:25.748603 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:25.748464    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba05e518-z46fv" event={"ID":"73e4cb78-cbd4-43aa-a9c7-5418323158db","Type":"ContainerDied","Data":"0bbc91104567ec028fe9bf983fa04158ad75da1016bc8f5f5f1d6a25415bb964"}
Oct 30 21:55:25.908886 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:25.908854    5087 scope.go:122] "RemoveContainer" containerID="b4be0fe9a1c26c8c5a78647c8cc349da3e905fb197d2733630dfadee01286828"
Oct 30 21:55:25.919035 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:25.919015    5087 scope.go:122] "RemoveContainer" containerID="cc5f22100064da7e6408b232086cd285fc5b4a3ccdf79a3506f08f329dab556b"
Oct 30 21:55:26.916736 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:26.916712    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba05e518-z46fv"
Oct 30 21:55:26.969486 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:26.969455    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73e4cb78-cbd4-43aa-a9c7-5418323158db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73e4cb78-cbd4-43aa-a9c7-5418323158db-maas-api-openshift-ca\") pod \"73e4cb78-cbd4-43aa-a9c7-5418323158db\" (UID: \"73e4cb78-cbd4-43aa-a9c7-5418323158db\") "
Oct 30 21:55:26.969650 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:26.969509    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73e4cb78-cbd4-43aa-a9c7-5418323158db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73e4cb78-cbd4-43aa-a9c7-5418323158db-maas-api-xks-ca\") pod \"73e4cb78-cbd4-43aa-a9c7-5418323158db\" (UID: \"73e4cb78-cbd4-43aa-a9c7-5418323158db\") "
Oct 30 21:55:26.969650 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:26.969603    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73e4cb78-cbd4-43aa-a9c7-5418323158db-kube-api-access-hst9z\" (UniqueName: \"kubernetes.io/projected/73e4cb78-cbd4-43aa-a9c7-5418323158db-kube-api-access-hst9z\") pod \"73e4cb78-cbd4-43aa-a9c7-5418323158db\" (UID: \"73e4cb78-cbd4-43aa-a9c7-5418323158db\") "
Oct 30 21:55:26.969861 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:26.969832    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e4cb78-cbd4-43aa-a9c7-5418323158db-maas-api-openshift-ca" pod "73e4cb78-cbd4-43aa-a9c7-5418323158db" (UID: "73e4cb78-cbd4-43aa-a9c7-5418323158db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:26.970134 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:26.970079    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73e4cb78-cbd4-43aa-a9c7-5418323158db-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:26.971748 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:26.971728    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73e4cb78-cbd4-43aa-a9c7-5418323158db-maas-api-xks-ca" pod "73e4cb78-cbd4-43aa-a9c7-5418323158db" (UID: "73e4cb78-cbd4-43aa-a9c7-5418323158db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:26.972089 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:26.972075    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e4cb78-cbd4-43aa-a9c7-5418323158db-kube-api-access-hst9z" pod "73e4cb78-cbd4-43aa-a9c7-5418323158db" (UID: "73e4cb78-cbd4-43aa-a9c7-5418323158db"). InnerVolumeSpecName "kube-api-access-hst9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:27.072447 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:27.071544    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73e4cb78-cbd4-43aa-a9c7-5418323158db-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:27.072447 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:27.071575    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hst9z\" (UniqueName: \"kubernetes.io/projected/73e4cb78-cbd4-43aa-a9c7-5418323158db-kube-api-access-hst9z\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:27.768404 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:27.768369    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba05e518-z46fv"
Oct 30 21:55:27.768582 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:27.768383    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ba05e518-z46fv" event={"ID":"73e4cb78-cbd4-43aa-a9c7-5418323158db","Type":"ContainerDied","Data":"bf0c416855528693eebd4b122a0edf045f987f8f2f834569d15e51bbf8bf91de"}
Oct 30 21:55:27.768582 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:27.768448    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf0c416855528693eebd4b122a0edf045f987f8f2f834569d15e51bbf8bf91de"
Oct 30 21:55:30.062129 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.062095    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe21caf8-29dhc"]
Oct 30 21:55:30.110702 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.110671    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe21caf8-29dhc"]
Oct 30 21:55:30.216021 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.215984    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"]
Oct 30 21:55:30.216661 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.216641    5087 cpu_manager.go:414] "containerMap: removing container" podUID="73e4cb78-cbd4-43aa-a9c7-5418323158db" containerName="revoke-keys"
Oct 30 21:55:30.216661 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.216662    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e4cb78-cbd4-43aa-a9c7-5418323158db" containerName="revoke-keys"
Oct 30 21:55:30.216845 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.216831    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="73e4cb78-cbd4-43aa-a9c7-5418323158db" containerName="revoke-keys"
Oct 30 21:55:30.220504 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.220472    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:30.283859 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.283824    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5966c85-72dkf"]
Oct 30 21:55:30.289130 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.289106    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5966c85-72dkf"
Oct 30 21:55:30.312004 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.311976    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"]
Oct 30 21:55:30.318293 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.318210    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-workload-socket\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:30.318293 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.318252    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-istio-envoy\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:30.318293 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.318290    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-workload-certs\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:30.318627 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.318459    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b411531-02da-4593-a357-1a15f782c76b-istio-token\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:30.318627 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.318504    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b411531-02da-4593-a357-1a15f782c76b-istiod-ca-cert\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:30.318627 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.318573    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-credential-socket\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:30.318627 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.318617    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-istio-data\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:30.318871 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.318640    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b411531-02da-4593-a357-1a15f782c76b-istio-podinfo\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:30.318871 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.318679    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxvfx\" (UniqueName: \"kubernetes.io/projected/1b411531-02da-4593-a357-1a15f782c76b-kube-api-access-gxvfx\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:30.339181 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.339152    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5966c85-72dkf"]
Oct 30 21:55:30.344555 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.344531    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:55:30.350080 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.350056    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:55:30.420103 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.420070    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-credential-socket\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:30.420265 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.420127    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-istio-data\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:30.420329 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.420305    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b411531-02da-4593-a357-1a15f782c76b-istio-podinfo\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:30.420448 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.420429    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd60c59-afc5-41f7-b751-121f62de5d74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5966c85-72dkf\" (UID: \"9dd60c59-afc5-41f7-b751-121f62de5d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5966c85-72dkf"
Oct 30 21:55:30.420516 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.420471    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-istio-data\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:30.420516 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.420473    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxvfx\" (UniqueName: \"kubernetes.io/projected/1b411531-02da-4593-a357-1a15f782c76b-kube-api-access-gxvfx\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:30.420623 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.420541    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-credential-socket\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:30.420623 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.420598    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwjmn\" (UniqueName: \"kubernetes.io/projected/9dd60c59-afc5-41f7-b751-121f62de5d74-kube-api-access-cwjmn\") pod \"maas-api-revoke-sub-a5966c85-72dkf\" (UID: \"9dd60c59-afc5-41f7-b751-121f62de5d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5966c85-72dkf"
Oct 30 21:55:30.420623 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.420622    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-workload-socket\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:30.420775 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.420638    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-istio-envoy\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:30.420775 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.420750    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-workload-certs\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:30.420847 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.420791    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd60c59-afc5-41f7-b751-121f62de5d74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5966c85-72dkf\" (UID: \"9dd60c59-afc5-41f7-b751-121f62de5d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5966c85-72dkf"
Oct 30 21:55:30.420884 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.420875    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b411531-02da-4593-a357-1a15f782c76b-istio-token\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:30.420925 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.420889    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-workload-socket\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:30.421019 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.420998    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b411531-02da-4593-a357-1a15f782c76b-istiod-ca-cert\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:30.421064 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.421050    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-workload-certs\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:30.421976 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.421950    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b411531-02da-4593-a357-1a15f782c76b-istiod-ca-cert\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:30.422962 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.422940    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-istio-envoy\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:30.423058 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.422995    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b411531-02da-4593-a357-1a15f782c76b-istio-podinfo\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:30.477509 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.477475    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxvfx\" (UniqueName: \"kubernetes.io/projected/1b411531-02da-4593-a357-1a15f782c76b-kube-api-access-gxvfx\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:30.478155 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.478129    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b411531-02da-4593-a357-1a15f782c76b-istio-token\") pod \"e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") " pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:30.521899 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.521873    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd60c59-afc5-41f7-b751-121f62de5d74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5966c85-72dkf\" (UID: \"9dd60c59-afc5-41f7-b751-121f62de5d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5966c85-72dkf"
Oct 30 21:55:30.522039 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.521936    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwjmn\" (UniqueName: \"kubernetes.io/projected/9dd60c59-afc5-41f7-b751-121f62de5d74-kube-api-access-cwjmn\") pod \"maas-api-revoke-sub-a5966c85-72dkf\" (UID: \"9dd60c59-afc5-41f7-b751-121f62de5d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5966c85-72dkf"
Oct 30 21:55:30.522039 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.521974    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd60c59-afc5-41f7-b751-121f62de5d74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5966c85-72dkf\" (UID: \"9dd60c59-afc5-41f7-b751-121f62de5d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5966c85-72dkf"
Oct 30 21:55:30.522627 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.522608    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd60c59-afc5-41f7-b751-121f62de5d74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5966c85-72dkf\" (UID: \"9dd60c59-afc5-41f7-b751-121f62de5d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5966c85-72dkf"
Oct 30 21:55:30.524198 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.524176    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd60c59-afc5-41f7-b751-121f62de5d74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5966c85-72dkf\" (UID: \"9dd60c59-afc5-41f7-b751-121f62de5d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5966c85-72dkf"
Oct 30 21:55:30.534461 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.534443    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:30.580088 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.580015    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwjmn\" (UniqueName: \"kubernetes.io/projected/9dd60c59-afc5-41f7-b751-121f62de5d74-kube-api-access-cwjmn\") pod \"maas-api-revoke-sub-a5966c85-72dkf\" (UID: \"9dd60c59-afc5-41f7-b751-121f62de5d74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5966c85-72dkf"
Oct 30 21:55:30.610926 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.609640    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5966c85-72dkf"
Oct 30 21:55:30.970208 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:30.970118    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"]
Oct 30 21:55:31.004342 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:55:31.004308    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b411531_02da_4593_a357_1a15f782c76b.slice/crio-15bdb23852a14e5b57d68d9e5f431d9bd9306c4d6541e1ce4798ba5f9618d55c WatchSource:0}: Error finding container 15bdb23852a14e5b57d68d9e5f431d9bd9306c4d6541e1ce4798ba5f9618d55c: Status 404 returned error can't find the container with id 15bdb23852a14e5b57d68d9e5f431d9bd9306c4d6541e1ce4798ba5f9618d55c
Oct 30 21:55:31.006864 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:31.006835    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:55:31.006948 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:31.006900    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:55:31.006948 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:31.006931    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:55:31.023527 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:31.023495    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5966c85-72dkf"]
Oct 30 21:55:31.030985 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:55:31.030957    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dd60c59_afc5_41f7_b751_121f62de5d74.slice/crio-4c977197be354aeba310a2df490f72f398c7fb3712243c2b46631e423d9e7957 WatchSource:0}: Error finding container 4c977197be354aeba310a2df490f72f398c7fb3712243c2b46631e423d9e7957: Status 404 returned error can't find the container with id 4c977197be354aeba310a2df490f72f398c7fb3712243c2b46631e423d9e7957
Oct 30 21:55:31.709939 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:31.709865    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f6ba80f-83df-4f15-84e0-a63607744fcb" path="/var/lib/kubelet/pods/4f6ba80f-83df-4f15-84e0-a63607744fcb/volumes"
Oct 30 21:55:31.803392 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:31.801799    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5966c85-72dkf" event={"ID":"9dd60c59-afc5-41f7-b751-121f62de5d74","Type":"ContainerStarted","Data":"1bc6d10da06ec1e1fac735331fa5705d852a07a1c01343a6ce3a9f39accc05c2"}
Oct 30 21:55:31.803392 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:31.801865    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5966c85-72dkf" event={"ID":"9dd60c59-afc5-41f7-b751-121f62de5d74","Type":"ContainerStarted","Data":"4c977197be354aeba310a2df490f72f398c7fb3712243c2b46631e423d9e7957"}
Oct 30 21:55:31.806372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:31.805201    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp" event={"ID":"1b411531-02da-4593-a357-1a15f782c76b","Type":"ContainerStarted","Data":"21d90f075be8baa5083465e6c5d42b5d825eb96fc2a8d69ca0fe0f99bf5f6ab7"}
Oct 30 21:55:31.806372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:31.805253    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp" event={"ID":"1b411531-02da-4593-a357-1a15f782c76b","Type":"ContainerStarted","Data":"15bdb23852a14e5b57d68d9e5f431d9bd9306c4d6541e1ce4798ba5f9618d55c"}
Oct 30 21:55:31.921021 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:31.920972    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5966c85-72dkf" podStartSLOduration=1.920958535 podStartE2EDuration="1.920958535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:31.918100149 +0000 UTC m=+1326.392604128" watchObservedRunningTime="2026-10-30 21:55:31.920958535 +0000 UTC m=+1326.395462514"
Oct 30 21:55:32.060368 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:32.060221    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp" podStartSLOduration=2.06019995 podStartE2EDuration="2.06019995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:32.049733662 +0000 UTC m=+1326.524237654" watchObservedRunningTime="2026-10-30 21:55:32.06019995 +0000 UTC m=+1326.534703931"
Oct 30 21:55:32.535524 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:32.535490    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:32.552373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:32.540457    5087 patch_prober.go:29] interesting pod/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 21:55:32.552373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:32.540513    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp" podUID="1b411531-02da-4593-a357-1a15f782c76b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 21:55:32.816504 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:32.816422    5087 generic.go:356] "Generic (PLEG): container finished" podID="9dd60c59-afc5-41f7-b751-121f62de5d74" containerID="1bc6d10da06ec1e1fac735331fa5705d852a07a1c01343a6ce3a9f39accc05c2" exitCode=0
Oct 30 21:55:32.816916 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:32.816507    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5966c85-72dkf" event={"ID":"9dd60c59-afc5-41f7-b751-121f62de5d74","Type":"ContainerDied","Data":"1bc6d10da06ec1e1fac735331fa5705d852a07a1c01343a6ce3a9f39accc05c2"}
Oct 30 21:55:33.535611 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:33.535575    5087 patch_prober.go:29] interesting pod/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 21:55:33.535794 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:33.535630    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp" podUID="1b411531-02da-4593-a357-1a15f782c76b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 21:55:34.084508 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:34.084434    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5966c85-72dkf"
Oct 30 21:55:34.096863 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:34.096837    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9dd60c59-afc5-41f7-b751-121f62de5d74-kube-api-access-cwjmn\" (UniqueName: \"kubernetes.io/projected/9dd60c59-afc5-41f7-b751-121f62de5d74-kube-api-access-cwjmn\") pod \"9dd60c59-afc5-41f7-b751-121f62de5d74\" (UID: \"9dd60c59-afc5-41f7-b751-121f62de5d74\") "
Oct 30 21:55:34.096993 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:34.096875    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9dd60c59-afc5-41f7-b751-121f62de5d74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd60c59-afc5-41f7-b751-121f62de5d74-maas-api-xks-ca\") pod \"9dd60c59-afc5-41f7-b751-121f62de5d74\" (UID: \"9dd60c59-afc5-41f7-b751-121f62de5d74\") "
Oct 30 21:55:34.096993 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:34.096933    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9dd60c59-afc5-41f7-b751-121f62de5d74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd60c59-afc5-41f7-b751-121f62de5d74-maas-api-openshift-ca\") pod \"9dd60c59-afc5-41f7-b751-121f62de5d74\" (UID: \"9dd60c59-afc5-41f7-b751-121f62de5d74\") "
Oct 30 21:55:34.097340 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:34.097317    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9dd60c59-afc5-41f7-b751-121f62de5d74-maas-api-openshift-ca" pod "9dd60c59-afc5-41f7-b751-121f62de5d74" (UID: "9dd60c59-afc5-41f7-b751-121f62de5d74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:34.099215 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:34.099187    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dd60c59-afc5-41f7-b751-121f62de5d74-maas-api-xks-ca" pod "9dd60c59-afc5-41f7-b751-121f62de5d74" (UID: "9dd60c59-afc5-41f7-b751-121f62de5d74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:34.099294 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:34.099246    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dd60c59-afc5-41f7-b751-121f62de5d74-kube-api-access-cwjmn" pod "9dd60c59-afc5-41f7-b751-121f62de5d74" (UID: "9dd60c59-afc5-41f7-b751-121f62de5d74"). InnerVolumeSpecName "kube-api-access-cwjmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:34.198125 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:34.198092    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9dd60c59-afc5-41f7-b751-121f62de5d74-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:34.198125 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:34.198122    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwjmn\" (UniqueName: \"kubernetes.io/projected/9dd60c59-afc5-41f7-b751-121f62de5d74-kube-api-access-cwjmn\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:34.198125 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:34.198132    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9dd60c59-afc5-41f7-b751-121f62de5d74-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:34.534989 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:34.534959    5087 patch_prober.go:29] interesting pod/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 21:55:34.535156 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:34.535010    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp" podUID="1b411531-02da-4593-a357-1a15f782c76b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 21:55:34.832092 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:34.832014    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5966c85-72dkf"
Oct 30 21:55:34.832092 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:34.832059    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5966c85-72dkf" event={"ID":"9dd60c59-afc5-41f7-b751-121f62de5d74","Type":"ContainerDied","Data":"4c977197be354aeba310a2df490f72f398c7fb3712243c2b46631e423d9e7957"}
Oct 30 21:55:34.832263 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:34.832147    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c977197be354aeba310a2df490f72f398c7fb3712243c2b46631e423d9e7957"
Oct 30 21:55:35.535868 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:35.535834    5087 patch_prober.go:29] interesting pod/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 21:55:35.536396 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:35.535900    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp" podUID="1b411531-02da-4593-a357-1a15f782c76b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 21:55:36.535151 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:36.535116    5087 patch_prober.go:29] interesting pod/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 21:55:36.535329 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:36.535200    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp" podUID="1b411531-02da-4593-a357-1a15f782c76b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 21:55:37.535045 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:37.535006    5087 patch_prober.go:29] interesting pod/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 30 21:55:37.535487 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:37.535080    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp" podUID="1b411531-02da-4593-a357-1a15f782c76b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 30 21:55:38.060544 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:38.060501    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d39ad088-f9dfn"]
Oct 30 21:55:38.106963 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:38.106929    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d39ad088-f9dfn"]
Oct 30 21:55:38.796025 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:38.794690    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:38.796025 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:38.795909    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:38.798570 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:38.798518    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 21:55:39.790401 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:39.783063    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15e0ff67-313d-4bad-93f1-c5924ea94c41" path="/var/lib/kubelet/pods/15e0ff67-313d-4bad-93f1-c5924ea94c41/volumes"
Oct 30 21:55:40.549469 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:40.549427    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:55:40.783503 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:40.782427    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:40.785144 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:40.784513    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:40.882523 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:40.882421    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:40.930937 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:40.930883    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:40.955889 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:40.955851    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:40.974600 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:40.974562    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:40.975880 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:40.975529    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:41.132513 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:41.114427    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dec2888-nkdmc"]
Oct 30 21:55:41.218075 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:41.217980    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dec2888-nkdmc"]
Oct 30 21:55:41.868546 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:41.868511    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af1c71a9-402d-4ac8-9cfa-a1ddf45f1274" path="/var/lib/kubelet/pods/af1c71a9-402d-4ac8-9cfa-a1ddf45f1274/volumes"
Oct 30 21:55:41.888525 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:41.888490    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 21:55:41.889086 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:41.889064    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:55:41.933473 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:41.933447    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:55:41.940568 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:41.940521    5087 patch_prober.go:29] interesting pod/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:55:41.940830 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:41.940795    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp" podUID="aed464f0-0b79-467a-b131-fa6944d01941" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:55:47.089903 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:47.089865    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c29418cd-x4x29"]
Oct 30 21:55:47.139245 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:47.139211    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c29418cd-x4x29"]
Oct 30 21:55:47.169082 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:55:47.169040    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:55:47.710946 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:47.710908    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65c43919-ddb4-413b-81aa-4c137227475f" path="/var/lib/kubelet/pods/65c43919-ddb4-413b-81aa-4c137227475f/volumes"
Oct 30 21:55:48.812294 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:48.812261    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f2e657d-r7x9f"]
Oct 30 21:55:48.812833 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:48.812819    5087 cpu_manager.go:414] "containerMap: removing container" podUID="9dd60c59-afc5-41f7-b751-121f62de5d74" containerName="revoke-keys"
Oct 30 21:55:48.812877 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:48.812834    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9dd60c59-afc5-41f7-b751-121f62de5d74" containerName="revoke-keys"
Oct 30 21:55:48.812961 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:48.812952    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="9dd60c59-afc5-41f7-b751-121f62de5d74" containerName="revoke-keys"
Oct 30 21:55:48.817904 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:48.817884    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e657d-r7x9f"
Oct 30 21:55:48.868569 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:48.868534    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce82a2ff-1f32-4ac4-ac06-3777474deddc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f2e657d-r7x9f\" (UID: \"ce82a2ff-1f32-4ac4-ac06-3777474deddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e657d-r7x9f"
Oct 30 21:55:48.868794 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:48.868680    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c5rg\" (UniqueName: \"kubernetes.io/projected/ce82a2ff-1f32-4ac4-ac06-3777474deddc-kube-api-access-5c5rg\") pod \"maas-api-revoke-sub-2f2e657d-r7x9f\" (UID: \"ce82a2ff-1f32-4ac4-ac06-3777474deddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e657d-r7x9f"
Oct 30 21:55:48.868794 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:48.868707    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce82a2ff-1f32-4ac4-ac06-3777474deddc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f2e657d-r7x9f\" (UID: \"ce82a2ff-1f32-4ac4-ac06-3777474deddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e657d-r7x9f"
Oct 30 21:55:48.874871 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:48.874845    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f2e657d-r7x9f"]
Oct 30 21:55:48.918514 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:48.918459    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:55:48.918514 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:48.918489    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:55:48.970026 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:48.969992    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce82a2ff-1f32-4ac4-ac06-3777474deddc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f2e657d-r7x9f\" (UID: \"ce82a2ff-1f32-4ac4-ac06-3777474deddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e657d-r7x9f"
Oct 30 21:55:48.970203 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:48.970109    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c5rg\" (UniqueName: \"kubernetes.io/projected/ce82a2ff-1f32-4ac4-ac06-3777474deddc-kube-api-access-5c5rg\") pod \"maas-api-revoke-sub-2f2e657d-r7x9f\" (UID: \"ce82a2ff-1f32-4ac4-ac06-3777474deddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e657d-r7x9f"
Oct 30 21:55:48.970203 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:48.970145    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce82a2ff-1f32-4ac4-ac06-3777474deddc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f2e657d-r7x9f\" (UID: \"ce82a2ff-1f32-4ac4-ac06-3777474deddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e657d-r7x9f"
Oct 30 21:55:48.970890 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:48.970808    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce82a2ff-1f32-4ac4-ac06-3777474deddc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f2e657d-r7x9f\" (UID: \"ce82a2ff-1f32-4ac4-ac06-3777474deddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e657d-r7x9f"
Oct 30 21:55:48.972560 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:48.972538    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce82a2ff-1f32-4ac4-ac06-3777474deddc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f2e657d-r7x9f\" (UID: \"ce82a2ff-1f32-4ac4-ac06-3777474deddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e657d-r7x9f"
Oct 30 21:55:49.028191 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:49.028161    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c5rg\" (UniqueName: \"kubernetes.io/projected/ce82a2ff-1f32-4ac4-ac06-3777474deddc-kube-api-access-5c5rg\") pod \"maas-api-revoke-sub-2f2e657d-r7x9f\" (UID: \"ce82a2ff-1f32-4ac4-ac06-3777474deddc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e657d-r7x9f"
Oct 30 21:55:49.131652 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:49.131580    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e657d-r7x9f"
Oct 30 21:55:49.515262 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:49.515235    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f2e657d-r7x9f"]
Oct 30 21:55:49.516267 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:55:49.516237    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce82a2ff_1f32_4ac4_ac06_3777474deddc.slice/crio-6563b3cd8364096ce2749df20ad7b933ba11cb2d574c5e51c79e8d8a0fffef9e WatchSource:0}: Error finding container 6563b3cd8364096ce2749df20ad7b933ba11cb2d574c5e51c79e8d8a0fffef9e: Status 404 returned error can't find the container with id 6563b3cd8364096ce2749df20ad7b933ba11cb2d574c5e51c79e8d8a0fffef9e
Oct 30 21:55:50.241710 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:50.241674    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e657d-r7x9f" event={"ID":"ce82a2ff-1f32-4ac4-ac06-3777474deddc","Type":"ContainerStarted","Data":"80f13c2d112b0d9bcf765886d6672a2b21f51c04abcda736614bdd5ba317ab2e"}
Oct 30 21:55:50.242099 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:50.241734    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e657d-r7x9f" event={"ID":"ce82a2ff-1f32-4ac4-ac06-3777474deddc","Type":"ContainerStarted","Data":"6563b3cd8364096ce2749df20ad7b933ba11cb2d574c5e51c79e8d8a0fffef9e"}
Oct 30 21:55:50.359710 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:50.359661    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e657d-r7x9f" podStartSLOduration=2.359647991 podStartE2EDuration="2.359647991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:50.355673923 +0000 UTC m=+1344.830177896" watchObservedRunningTime="2026-10-30 21:55:50.359647991 +0000 UTC m=+1344.834151969"
Oct 30 21:55:50.504852 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:50.504777    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e86549f5-g8j6m"]
Oct 30 21:55:50.509120 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:50.509102    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86549f5-g8j6m"
Oct 30 21:55:50.566632 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:50.566608    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e86549f5-g8j6m"]
Oct 30 21:55:50.592229 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:50.592196    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84d9b\" (UniqueName: \"kubernetes.io/projected/ae6565ae-cd62-4e3b-a95b-a658fbf78892-kube-api-access-84d9b\") pod \"maas-api-revoke-sub-e86549f5-g8j6m\" (UID: \"ae6565ae-cd62-4e3b-a95b-a658fbf78892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86549f5-g8j6m"
Oct 30 21:55:50.592431 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:50.592270    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae6565ae-cd62-4e3b-a95b-a658fbf78892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e86549f5-g8j6m\" (UID: \"ae6565ae-cd62-4e3b-a95b-a658fbf78892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86549f5-g8j6m"
Oct 30 21:55:50.592518 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:50.592470    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae6565ae-cd62-4e3b-a95b-a658fbf78892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e86549f5-g8j6m\" (UID: \"ae6565ae-cd62-4e3b-a95b-a658fbf78892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86549f5-g8j6m"
Oct 30 21:55:50.693422 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:50.693392    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84d9b\" (UniqueName: \"kubernetes.io/projected/ae6565ae-cd62-4e3b-a95b-a658fbf78892-kube-api-access-84d9b\") pod \"maas-api-revoke-sub-e86549f5-g8j6m\" (UID: \"ae6565ae-cd62-4e3b-a95b-a658fbf78892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86549f5-g8j6m"
Oct 30 21:55:50.693643 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:50.693477    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae6565ae-cd62-4e3b-a95b-a658fbf78892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e86549f5-g8j6m\" (UID: \"ae6565ae-cd62-4e3b-a95b-a658fbf78892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86549f5-g8j6m"
Oct 30 21:55:50.693643 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:50.693556    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae6565ae-cd62-4e3b-a95b-a658fbf78892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e86549f5-g8j6m\" (UID: \"ae6565ae-cd62-4e3b-a95b-a658fbf78892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86549f5-g8j6m"
Oct 30 21:55:50.694145 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:50.694119    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae6565ae-cd62-4e3b-a95b-a658fbf78892-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e86549f5-g8j6m\" (UID: \"ae6565ae-cd62-4e3b-a95b-a658fbf78892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86549f5-g8j6m"
Oct 30 21:55:50.696095 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:50.696071    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae6565ae-cd62-4e3b-a95b-a658fbf78892-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e86549f5-g8j6m\" (UID: \"ae6565ae-cd62-4e3b-a95b-a658fbf78892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86549f5-g8j6m"
Oct 30 21:55:50.761153 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:50.761082    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84d9b\" (UniqueName: \"kubernetes.io/projected/ae6565ae-cd62-4e3b-a95b-a658fbf78892-kube-api-access-84d9b\") pod \"maas-api-revoke-sub-e86549f5-g8j6m\" (UID: \"ae6565ae-cd62-4e3b-a95b-a658fbf78892\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86549f5-g8j6m"
Oct 30 21:55:50.822827 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:50.822798    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86549f5-g8j6m"
Oct 30 21:55:51.202645 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:51.202615    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e86549f5-g8j6m"]
Oct 30 21:55:51.203878 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:55:51.203850    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6565ae_cd62_4e3b_a95b_a658fbf78892.slice/crio-34b368596c5a9ab2dc4d8b9322858ea273ce7bff9a394568241c9c90cb449371 WatchSource:0}: Error finding container 34b368596c5a9ab2dc4d8b9322858ea273ce7bff9a394568241c9c90cb449371: Status 404 returned error can't find the container with id 34b368596c5a9ab2dc4d8b9322858ea273ce7bff9a394568241c9c90cb449371
Oct 30 21:55:51.250417 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:51.250381    5087 generic.go:356] "Generic (PLEG): container finished" podID="ce82a2ff-1f32-4ac4-ac06-3777474deddc" containerID="80f13c2d112b0d9bcf765886d6672a2b21f51c04abcda736614bdd5ba317ab2e" exitCode=0
Oct 30 21:55:51.250777 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:51.250441    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e657d-r7x9f" event={"ID":"ce82a2ff-1f32-4ac4-ac06-3777474deddc","Type":"ContainerDied","Data":"80f13c2d112b0d9bcf765886d6672a2b21f51c04abcda736614bdd5ba317ab2e"}
Oct 30 21:55:51.252146 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:51.252120    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86549f5-g8j6m" event={"ID":"ae6565ae-cd62-4e3b-a95b-a658fbf78892","Type":"ContainerStarted","Data":"34b368596c5a9ab2dc4d8b9322858ea273ce7bff9a394568241c9c90cb449371"}
Oct 30 21:55:52.051239 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:52.051205    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9083dfa4-8t5bs"]
Oct 30 21:55:52.057100 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:52.057069    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9083dfa4-8t5bs"]
Oct 30 21:55:52.262009 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:52.261980    5087 generic.go:356] "Generic (PLEG): container finished" podID="ae6565ae-cd62-4e3b-a95b-a658fbf78892" containerID="09d5af20382a619a1f3eff52c7b9c7319105f54b6704b99f2af699a5da842844" exitCode=0
Oct 30 21:55:52.262434 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:52.262061    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86549f5-g8j6m" event={"ID":"ae6565ae-cd62-4e3b-a95b-a658fbf78892","Type":"ContainerDied","Data":"09d5af20382a619a1f3eff52c7b9c7319105f54b6704b99f2af699a5da842844"}
Oct 30 21:55:52.417878 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:52.417854    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e657d-r7x9f"
Oct 30 21:55:52.516589 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:52.516557    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce82a2ff-1f32-4ac4-ac06-3777474deddc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce82a2ff-1f32-4ac4-ac06-3777474deddc-maas-api-xks-ca\") pod \"ce82a2ff-1f32-4ac4-ac06-3777474deddc\" (UID: \"ce82a2ff-1f32-4ac4-ac06-3777474deddc\") "
Oct 30 21:55:52.516748 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:52.516650    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce82a2ff-1f32-4ac4-ac06-3777474deddc-kube-api-access-5c5rg\" (UniqueName: \"kubernetes.io/projected/ce82a2ff-1f32-4ac4-ac06-3777474deddc-kube-api-access-5c5rg\") pod \"ce82a2ff-1f32-4ac4-ac06-3777474deddc\" (UID: \"ce82a2ff-1f32-4ac4-ac06-3777474deddc\") "
Oct 30 21:55:52.516748 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:52.516678    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce82a2ff-1f32-4ac4-ac06-3777474deddc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce82a2ff-1f32-4ac4-ac06-3777474deddc-maas-api-openshift-ca\") pod \"ce82a2ff-1f32-4ac4-ac06-3777474deddc\" (UID: \"ce82a2ff-1f32-4ac4-ac06-3777474deddc\") "
Oct 30 21:55:52.517097 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:52.517069    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce82a2ff-1f32-4ac4-ac06-3777474deddc-maas-api-openshift-ca" pod "ce82a2ff-1f32-4ac4-ac06-3777474deddc" (UID: "ce82a2ff-1f32-4ac4-ac06-3777474deddc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:52.517419 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:52.517305    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce82a2ff-1f32-4ac4-ac06-3777474deddc-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:52.518862 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:52.518834    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce82a2ff-1f32-4ac4-ac06-3777474deddc-maas-api-xks-ca" pod "ce82a2ff-1f32-4ac4-ac06-3777474deddc" (UID: "ce82a2ff-1f32-4ac4-ac06-3777474deddc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:52.518943 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:52.518870    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce82a2ff-1f32-4ac4-ac06-3777474deddc-kube-api-access-5c5rg" pod "ce82a2ff-1f32-4ac4-ac06-3777474deddc" (UID: "ce82a2ff-1f32-4ac4-ac06-3777474deddc"). InnerVolumeSpecName "kube-api-access-5c5rg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:52.618245 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:52.618173    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce82a2ff-1f32-4ac4-ac06-3777474deddc-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:52.618245 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:52.618201    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c5rg\" (UniqueName: \"kubernetes.io/projected/ce82a2ff-1f32-4ac4-ac06-3777474deddc-kube-api-access-5c5rg\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:52.953866 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:52.953787    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd6c31e4-5lsd9"]
Oct 30 21:55:52.954725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:52.954696    5087 cpu_manager.go:414] "containerMap: removing container" podUID="ce82a2ff-1f32-4ac4-ac06-3777474deddc" containerName="revoke-keys"
Oct 30 21:55:52.954725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:52.954720    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce82a2ff-1f32-4ac4-ac06-3777474deddc" containerName="revoke-keys"
Oct 30 21:55:52.954926 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:52.954849    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce82a2ff-1f32-4ac4-ac06-3777474deddc" containerName="revoke-keys"
Oct 30 21:55:52.959265 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:52.959242    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6c31e4-5lsd9"
Oct 30 21:55:53.027429 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:53.026998    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c436ac2f-8f04-4af3-92f5-22968150af31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd6c31e4-5lsd9\" (UID: \"c436ac2f-8f04-4af3-92f5-22968150af31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6c31e4-5lsd9"
Oct 30 21:55:53.027429 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:53.027081    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b89w8\" (UniqueName: \"kubernetes.io/projected/c436ac2f-8f04-4af3-92f5-22968150af31-kube-api-access-b89w8\") pod \"maas-api-revoke-sub-dd6c31e4-5lsd9\" (UID: \"c436ac2f-8f04-4af3-92f5-22968150af31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6c31e4-5lsd9"
Oct 30 21:55:53.027707 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:53.027443    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c436ac2f-8f04-4af3-92f5-22968150af31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd6c31e4-5lsd9\" (UID: \"c436ac2f-8f04-4af3-92f5-22968150af31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6c31e4-5lsd9"
Oct 30 21:55:53.042411 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:53.030171    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd6c31e4-5lsd9"]
Oct 30 21:55:53.128820 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:53.128784    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c436ac2f-8f04-4af3-92f5-22968150af31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd6c31e4-5lsd9\" (UID: \"c436ac2f-8f04-4af3-92f5-22968150af31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6c31e4-5lsd9"
Oct 30 21:55:53.129013 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:53.128864    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c436ac2f-8f04-4af3-92f5-22968150af31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd6c31e4-5lsd9\" (UID: \"c436ac2f-8f04-4af3-92f5-22968150af31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6c31e4-5lsd9"
Oct 30 21:55:53.129013 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:53.128926    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b89w8\" (UniqueName: \"kubernetes.io/projected/c436ac2f-8f04-4af3-92f5-22968150af31-kube-api-access-b89w8\") pod \"maas-api-revoke-sub-dd6c31e4-5lsd9\" (UID: \"c436ac2f-8f04-4af3-92f5-22968150af31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6c31e4-5lsd9"
Oct 30 21:55:53.129825 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:53.129795    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c436ac2f-8f04-4af3-92f5-22968150af31-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd6c31e4-5lsd9\" (UID: \"c436ac2f-8f04-4af3-92f5-22968150af31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6c31e4-5lsd9"
Oct 30 21:55:53.132170 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:53.132147    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c436ac2f-8f04-4af3-92f5-22968150af31-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd6c31e4-5lsd9\" (UID: \"c436ac2f-8f04-4af3-92f5-22968150af31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6c31e4-5lsd9"
Oct 30 21:55:53.187162 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:53.187132    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b89w8\" (UniqueName: \"kubernetes.io/projected/c436ac2f-8f04-4af3-92f5-22968150af31-kube-api-access-b89w8\") pod \"maas-api-revoke-sub-dd6c31e4-5lsd9\" (UID: \"c436ac2f-8f04-4af3-92f5-22968150af31\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6c31e4-5lsd9"
Oct 30 21:55:53.271382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:53.271324    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e657d-r7x9f"
Oct 30 21:55:53.271382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:53.271359    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f2e657d-r7x9f" event={"ID":"ce82a2ff-1f32-4ac4-ac06-3777474deddc","Type":"ContainerDied","Data":"6563b3cd8364096ce2749df20ad7b933ba11cb2d574c5e51c79e8d8a0fffef9e"}
Oct 30 21:55:53.271845 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:53.271411    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6563b3cd8364096ce2749df20ad7b933ba11cb2d574c5e51c79e8d8a0fffef9e"
Oct 30 21:55:53.278827 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:53.278801    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6c31e4-5lsd9"
Oct 30 21:55:53.468757 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:53.468734    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86549f5-g8j6m"
Oct 30 21:55:53.533558 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:53.533488    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae6565ae-cd62-4e3b-a95b-a658fbf78892-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae6565ae-cd62-4e3b-a95b-a658fbf78892-maas-api-openshift-ca\") pod \"ae6565ae-cd62-4e3b-a95b-a658fbf78892\" (UID: \"ae6565ae-cd62-4e3b-a95b-a658fbf78892\") "
Oct 30 21:55:53.533717 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:53.533572    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae6565ae-cd62-4e3b-a95b-a658fbf78892-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae6565ae-cd62-4e3b-a95b-a658fbf78892-maas-api-xks-ca\") pod \"ae6565ae-cd62-4e3b-a95b-a658fbf78892\" (UID: \"ae6565ae-cd62-4e3b-a95b-a658fbf78892\") "
Oct 30 21:55:53.533717 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:53.533611    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae6565ae-cd62-4e3b-a95b-a658fbf78892-kube-api-access-84d9b\" (UniqueName: \"kubernetes.io/projected/ae6565ae-cd62-4e3b-a95b-a658fbf78892-kube-api-access-84d9b\") pod \"ae6565ae-cd62-4e3b-a95b-a658fbf78892\" (UID: \"ae6565ae-cd62-4e3b-a95b-a658fbf78892\") "
Oct 30 21:55:53.533870 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:53.533846    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae6565ae-cd62-4e3b-a95b-a658fbf78892-maas-api-openshift-ca" pod "ae6565ae-cd62-4e3b-a95b-a658fbf78892" (UID: "ae6565ae-cd62-4e3b-a95b-a658fbf78892"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:53.534218 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:53.534197    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae6565ae-cd62-4e3b-a95b-a658fbf78892-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:53.535843 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:53.535819    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae6565ae-cd62-4e3b-a95b-a658fbf78892-maas-api-xks-ca" pod "ae6565ae-cd62-4e3b-a95b-a658fbf78892" (UID: "ae6565ae-cd62-4e3b-a95b-a658fbf78892"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:53.535941 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:53.535895    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6565ae-cd62-4e3b-a95b-a658fbf78892-kube-api-access-84d9b" pod "ae6565ae-cd62-4e3b-a95b-a658fbf78892" (UID: "ae6565ae-cd62-4e3b-a95b-a658fbf78892"). InnerVolumeSpecName "kube-api-access-84d9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:53.636063 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:53.636030    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae6565ae-cd62-4e3b-a95b-a658fbf78892-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:53.636063 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:53.636061    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84d9b\" (UniqueName: \"kubernetes.io/projected/ae6565ae-cd62-4e3b-a95b-a658fbf78892-kube-api-access-84d9b\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:53.672027 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:55:53.671999    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc436ac2f_8f04_4af3_92f5_22968150af31.slice/crio-dca8c07161b97dd53c338e7371ae2d3f27047b6c37415be63e50a9df0574c633 WatchSource:0}: Error finding container dca8c07161b97dd53c338e7371ae2d3f27047b6c37415be63e50a9df0574c633: Status 404 returned error can't find the container with id dca8c07161b97dd53c338e7371ae2d3f27047b6c37415be63e50a9df0574c633
Oct 30 21:55:53.672027 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:53.672005    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd6c31e4-5lsd9"]
Oct 30 21:55:53.714479 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:53.714454    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fdca4be-6f86-45e3-bbd3-3e78a283a6f5" path="/var/lib/kubelet/pods/8fdca4be-6f86-45e3-bbd3-3e78a283a6f5/volumes"
Oct 30 21:55:54.282132 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:54.282090    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86549f5-g8j6m" event={"ID":"ae6565ae-cd62-4e3b-a95b-a658fbf78892","Type":"ContainerDied","Data":"34b368596c5a9ab2dc4d8b9322858ea273ce7bff9a394568241c9c90cb449371"}
Oct 30 21:55:54.282132 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:54.282118    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e86549f5-g8j6m"
Oct 30 21:55:54.282665 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:54.282183    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34b368596c5a9ab2dc4d8b9322858ea273ce7bff9a394568241c9c90cb449371"
Oct 30 21:55:54.284407 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:54.284372    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6c31e4-5lsd9" event={"ID":"c436ac2f-8f04-4af3-92f5-22968150af31","Type":"ContainerStarted","Data":"c97b0f93480e7ee36f71c718131946d665273e4df35b0e2246a021785eb0a33d"}
Oct 30 21:55:54.284568 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:54.284452    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6c31e4-5lsd9" event={"ID":"c436ac2f-8f04-4af3-92f5-22968150af31","Type":"ContainerStarted","Data":"dca8c07161b97dd53c338e7371ae2d3f27047b6c37415be63e50a9df0574c633"}
Oct 30 21:55:54.614830 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:54.614781    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6c31e4-5lsd9" podStartSLOduration=2.614765652 podStartE2EDuration="2.614765652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:54.446860196 +0000 UTC m=+1348.921364175" watchObservedRunningTime="2026-10-30 21:55:54.614765652 +0000 UTC m=+1349.089269630"
Oct 30 21:55:55.293113 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:55.293077    5087 generic.go:356] "Generic (PLEG): container finished" podID="c436ac2f-8f04-4af3-92f5-22968150af31" containerID="c97b0f93480e7ee36f71c718131946d665273e4df35b0e2246a021785eb0a33d" exitCode=0
Oct 30 21:55:55.293674 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:55.293159    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6c31e4-5lsd9" event={"ID":"c436ac2f-8f04-4af3-92f5-22968150af31","Type":"ContainerDied","Data":"c97b0f93480e7ee36f71c718131946d665273e4df35b0e2246a021785eb0a33d"}
Oct 30 21:55:56.459147 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:56.459121    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6c31e4-5lsd9"
Oct 30 21:55:56.580149 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:56.580069    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c436ac2f-8f04-4af3-92f5-22968150af31-kube-api-access-b89w8\" (UniqueName: \"kubernetes.io/projected/c436ac2f-8f04-4af3-92f5-22968150af31-kube-api-access-b89w8\") pod \"c436ac2f-8f04-4af3-92f5-22968150af31\" (UID: \"c436ac2f-8f04-4af3-92f5-22968150af31\") "
Oct 30 21:55:56.580149 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:56.580136    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c436ac2f-8f04-4af3-92f5-22968150af31-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c436ac2f-8f04-4af3-92f5-22968150af31-maas-api-openshift-ca\") pod \"c436ac2f-8f04-4af3-92f5-22968150af31\" (UID: \"c436ac2f-8f04-4af3-92f5-22968150af31\") "
Oct 30 21:55:56.580432 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:56.580231    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c436ac2f-8f04-4af3-92f5-22968150af31-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c436ac2f-8f04-4af3-92f5-22968150af31-maas-api-xks-ca\") pod \"c436ac2f-8f04-4af3-92f5-22968150af31\" (UID: \"c436ac2f-8f04-4af3-92f5-22968150af31\") "
Oct 30 21:55:56.580564 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:56.580542    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c436ac2f-8f04-4af3-92f5-22968150af31-maas-api-openshift-ca" pod "c436ac2f-8f04-4af3-92f5-22968150af31" (UID: "c436ac2f-8f04-4af3-92f5-22968150af31"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:55:56.580759 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:56.580742    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c436ac2f-8f04-4af3-92f5-22968150af31-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:56.582463 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:56.582442    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c436ac2f-8f04-4af3-92f5-22968150af31-maas-api-xks-ca" pod "c436ac2f-8f04-4af3-92f5-22968150af31" (UID: "c436ac2f-8f04-4af3-92f5-22968150af31"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:55:56.582528 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:56.582466    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c436ac2f-8f04-4af3-92f5-22968150af31-kube-api-access-b89w8" pod "c436ac2f-8f04-4af3-92f5-22968150af31" (UID: "c436ac2f-8f04-4af3-92f5-22968150af31"). InnerVolumeSpecName "kube-api-access-b89w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:55:56.682526 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:56.682490    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c436ac2f-8f04-4af3-92f5-22968150af31-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:56.682526 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:56.682518    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b89w8\" (UniqueName: \"kubernetes.io/projected/c436ac2f-8f04-4af3-92f5-22968150af31-kube-api-access-b89w8\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:55:57.307764 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:57.307737    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6c31e4-5lsd9"
Oct 30 21:55:57.307936 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:57.307779    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd6c31e4-5lsd9" event={"ID":"c436ac2f-8f04-4af3-92f5-22968150af31","Type":"ContainerDied","Data":"dca8c07161b97dd53c338e7371ae2d3f27047b6c37415be63e50a9df0574c633"}
Oct 30 21:55:57.307936 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:57.307824    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dca8c07161b97dd53c338e7371ae2d3f27047b6c37415be63e50a9df0574c633"
Oct 30 21:55:59.064764 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.064732    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8af1a54-vh75s"]
Oct 30 21:55:59.112073 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.112036    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8af1a54-vh75s"]
Oct 30 21:55:59.535468 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.535436    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"]
Oct 30 21:55:59.536028 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.536012    5087 cpu_manager.go:414] "containerMap: removing container" podUID="ae6565ae-cd62-4e3b-a95b-a658fbf78892" containerName="revoke-keys"
Oct 30 21:55:59.536077 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.536029    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae6565ae-cd62-4e3b-a95b-a658fbf78892" containerName="revoke-keys"
Oct 30 21:55:59.536077 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.536063    5087 cpu_manager.go:414] "containerMap: removing container" podUID="c436ac2f-8f04-4af3-92f5-22968150af31" containerName="revoke-keys"
Oct 30 21:55:59.536077 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.536068    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c436ac2f-8f04-4af3-92f5-22968150af31" containerName="revoke-keys"
Oct 30 21:55:59.536218 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.536174    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae6565ae-cd62-4e3b-a95b-a658fbf78892" containerName="revoke-keys"
Oct 30 21:55:59.536218 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.536186    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="c436ac2f-8f04-4af3-92f5-22968150af31" containerName="revoke-keys"
Oct 30 21:55:59.539878 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.539862    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:55:59.585355 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.585328    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6afc4d44-gdqgr"]
Oct 30 21:55:59.589504 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.589486    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6afc4d44-gdqgr"
Oct 30 21:55:59.618004 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.617969    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istio-data\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:55:59.618128 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.618047    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istio-podinfo\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:55:59.618128 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.618082    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ad2bc8-ec58-4b55-8c49-869450d196f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6afc4d44-gdqgr\" (UID: \"30ad2bc8-ec58-4b55-8c49-869450d196f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6afc4d44-gdqgr"
Oct 30 21:55:59.618245 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.618157    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istio-envoy\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:55:59.618245 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.618192    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istiod-ca-cert\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:55:59.618245 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.618208    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ad2bc8-ec58-4b55-8c49-869450d196f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6afc4d44-gdqgr\" (UID: \"30ad2bc8-ec58-4b55-8c49-869450d196f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6afc4d44-gdqgr"
Oct 30 21:55:59.618416 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.618288    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istio-token\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:55:59.618416 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.618364    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-workload-socket\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:55:59.618416 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.618386    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-workload-certs\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:55:59.618565 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.618512    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdv69\" (UniqueName: \"kubernetes.io/projected/30ad2bc8-ec58-4b55-8c49-869450d196f6-kube-api-access-kdv69\") pod \"maas-api-revoke-sub-6afc4d44-gdqgr\" (UID: \"30ad2bc8-ec58-4b55-8c49-869450d196f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6afc4d44-gdqgr"
Oct 30 21:55:59.618619 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.618563    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctjwd\" (UniqueName: \"kubernetes.io/projected/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-kube-api-access-ctjwd\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:55:59.618683 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.618665    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-credential-socket\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:55:59.640447 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.640416    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"]
Oct 30 21:55:59.646867 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.646839    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:55:59.647012 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.646993    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:55:59.668822 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.668797    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6afc4d44-gdqgr"]
Oct 30 21:55:59.715018 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.714984    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77a84962-471b-4ce2-9561-ceb900285afd" path="/var/lib/kubelet/pods/77a84962-471b-4ce2-9561-ceb900285afd/volumes"
Oct 30 21:55:59.719684 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.719662    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdv69\" (UniqueName: \"kubernetes.io/projected/30ad2bc8-ec58-4b55-8c49-869450d196f6-kube-api-access-kdv69\") pod \"maas-api-revoke-sub-6afc4d44-gdqgr\" (UID: \"30ad2bc8-ec58-4b55-8c49-869450d196f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6afc4d44-gdqgr"
Oct 30 21:55:59.719779 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.719724    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctjwd\" (UniqueName: \"kubernetes.io/projected/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-kube-api-access-ctjwd\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:55:59.719779 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.719759    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-credential-socket\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:55:59.719881 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.719865    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istio-data\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:55:59.719936 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.719893    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istio-podinfo\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:55:59.720017 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.719997    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ad2bc8-ec58-4b55-8c49-869450d196f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6afc4d44-gdqgr\" (UID: \"30ad2bc8-ec58-4b55-8c49-869450d196f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6afc4d44-gdqgr"
Oct 30 21:55:59.720080 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.720048    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istio-envoy\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:55:59.720141 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.720104    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istiod-ca-cert\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:55:59.720141 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.720132    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ad2bc8-ec58-4b55-8c49-869450d196f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6afc4d44-gdqgr\" (UID: \"30ad2bc8-ec58-4b55-8c49-869450d196f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6afc4d44-gdqgr"
Oct 30 21:55:59.720243 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.720171    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istio-token\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:55:59.720304 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.720261    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-workload-socket\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:55:59.720304 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.720279    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-credential-socket\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:55:59.720433 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.720308    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-workload-certs\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:55:59.720433 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.720306    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istio-data\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:55:59.720650 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.720607    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-workload-socket\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:55:59.720883 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.720861    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-workload-certs\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:55:59.720955 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.720911    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ad2bc8-ec58-4b55-8c49-869450d196f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6afc4d44-gdqgr\" (UID: \"30ad2bc8-ec58-4b55-8c49-869450d196f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6afc4d44-gdqgr"
Oct 30 21:55:59.721014 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.720980    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istiod-ca-cert\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:55:59.723076 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.723052    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istio-envoy\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:55:59.723168 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.723135    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ad2bc8-ec58-4b55-8c49-869450d196f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6afc4d44-gdqgr\" (UID: \"30ad2bc8-ec58-4b55-8c49-869450d196f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6afc4d44-gdqgr"
Oct 30 21:55:59.723168 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.723141    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istio-podinfo\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:55:59.777400 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.777340    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdv69\" (UniqueName: \"kubernetes.io/projected/30ad2bc8-ec58-4b55-8c49-869450d196f6-kube-api-access-kdv69\") pod \"maas-api-revoke-sub-6afc4d44-gdqgr\" (UID: \"30ad2bc8-ec58-4b55-8c49-869450d196f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6afc4d44-gdqgr"
Oct 30 21:55:59.777765 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.777742    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istio-token\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:55:59.777868 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.777854    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctjwd\" (UniqueName: \"kubernetes.io/projected/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-kube-api-access-ctjwd\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:55:59.854543 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.854473    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:55:59.903327 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:55:59.903296    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6afc4d44-gdqgr"
Oct 30 21:56:00.094472 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.094422    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"]
Oct 30 21:56:00.100234 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.100213    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:00.125848 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.125823    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/727b252d-ad3e-4e43-9fb8-471fd2957231-istiod-ca-cert\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:00.125971 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.125889    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/727b252d-ad3e-4e43-9fb8-471fd2957231-istio-podinfo\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:00.126044 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.126018    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/727b252d-ad3e-4e43-9fb8-471fd2957231-istio-token\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:00.126171 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.126152    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfk2t\" (UniqueName: \"kubernetes.io/projected/727b252d-ad3e-4e43-9fb8-471fd2957231-kube-api-access-dfk2t\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:00.126214 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.126199    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-istio-data\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:00.126310 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.126290    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-workload-certs\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:00.126454 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.126436    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-workload-socket\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:00.126522 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.126472    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-istio-envoy\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:00.126571 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.126553    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-credential-socket\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:00.171948 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.171918    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"]
Oct 30 21:56:00.228507 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.228469    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/727b252d-ad3e-4e43-9fb8-471fd2957231-istiod-ca-cert\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:00.228703 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.228552    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/727b252d-ad3e-4e43-9fb8-471fd2957231-istio-podinfo\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:00.228703 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.228611    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/727b252d-ad3e-4e43-9fb8-471fd2957231-istio-token\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:00.228827 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.228702    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfk2t\" (UniqueName: \"kubernetes.io/projected/727b252d-ad3e-4e43-9fb8-471fd2957231-kube-api-access-dfk2t\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:00.228827 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.228740    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-istio-data\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:00.228827 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.228777    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-workload-certs\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:00.228986 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.228839    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-workload-socket\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:00.228986 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.228871    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-istio-envoy\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:00.228986 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.228921    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-credential-socket\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:00.229431 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.229335    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-credential-socket\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:00.229431 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.229399    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-workload-certs\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:00.229664 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.229649    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/727b252d-ad3e-4e43-9fb8-471fd2957231-istiod-ca-cert\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:00.229784 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.229761    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-istio-data\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:00.229858 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.229817    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-workload-socket\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:00.231598 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.231575    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/727b252d-ad3e-4e43-9fb8-471fd2957231-istio-podinfo\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:00.232471 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.232452    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-istio-envoy\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:00.286932 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.286902    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/727b252d-ad3e-4e43-9fb8-471fd2957231-istio-token\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:00.287467 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.287445    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfk2t\" (UniqueName: \"kubernetes.io/projected/727b252d-ad3e-4e43-9fb8-471fd2957231-kube-api-access-dfk2t\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:00.346207 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:56:00.346142    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ad2bc8_ec58_4b55_8c49_869450d196f6.slice/crio-4ba76ab3bf0e17cccdd6bce1e32944d81cdcc7c614d321f4c61b6b7f6132e0e1 WatchSource:0}: Error finding container 4ba76ab3bf0e17cccdd6bce1e32944d81cdcc7c614d321f4c61b6b7f6132e0e1: Status 404 returned error can't find the container with id 4ba76ab3bf0e17cccdd6bce1e32944d81cdcc7c614d321f4c61b6b7f6132e0e1
Oct 30 21:56:00.346207 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.346184    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6afc4d44-gdqgr"]
Oct 30 21:56:00.352149 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:56:00.352120    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db8cae8_a410_4c25_bdec_cee65fb1a8ae.slice/crio-91949efa101975e9eaef2bcaf421d5847138e0ba841e2aef3a5f91bb4daf2d45 WatchSource:0}: Error finding container 91949efa101975e9eaef2bcaf421d5847138e0ba841e2aef3a5f91bb4daf2d45: Status 404 returned error can't find the container with id 91949efa101975e9eaef2bcaf421d5847138e0ba841e2aef3a5f91bb4daf2d45
Oct 30 21:56:00.352733 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.352713    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"]
Oct 30 21:56:00.357875 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.357840    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:56:00.357951 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.357918    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:56:00.357993 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.357960    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:56:00.417432 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.417410    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:00.739626 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.739589    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"]
Oct 30 21:56:00.749594 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.749566    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:00.823139 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.814043    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"]
Oct 30 21:56:00.823960 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.823909    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"]
Oct 30 21:56:00.827264 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:56:00.827236    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod727b252d_ad3e_4e43_9fb8_471fd2957231.slice/crio-79367c6f291c911af631d88b7ea6c6f99c0bf9c0aa7b427374b528248873d2f4 WatchSource:0}: Error finding container 79367c6f291c911af631d88b7ea6c6f99c0bf9c0aa7b427374b528248873d2f4: Status 404 returned error can't find the container with id 79367c6f291c911af631d88b7ea6c6f99c0bf9c0aa7b427374b528248873d2f4
Oct 30 21:56:00.830145 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.830105    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:56:00.830241 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.830187    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:56:00.830241 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.830228    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:56:00.844076 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.844047    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f891215-2cc6-4b56-8ba2-68687e58f21c-istiod-ca-cert\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:00.845564 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.845535    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-istio-data\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:00.845688 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.845584    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f891215-2cc6-4b56-8ba2-68687e58f21c-istio-podinfo\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:00.845779 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.845723    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-workload-socket\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:00.845862 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.845809    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-credential-socket\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:00.845862 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.845840    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-workload-certs\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:00.845978 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.845963    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f891215-2cc6-4b56-8ba2-68687e58f21c-istio-token\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:00.846052 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.846041    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-istio-envoy\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:00.846112 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.846079    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zlwm\" (UniqueName: \"kubernetes.io/projected/6f891215-2cc6-4b56-8ba2-68687e58f21c-kube-api-access-8zlwm\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:00.956371 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.956042    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f891215-2cc6-4b56-8ba2-68687e58f21c-istio-token\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:00.956371 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.956181    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-istio-envoy\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:00.956371 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.956238    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zlwm\" (UniqueName: \"kubernetes.io/projected/6f891215-2cc6-4b56-8ba2-68687e58f21c-kube-api-access-8zlwm\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:00.956371 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.956299    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f891215-2cc6-4b56-8ba2-68687e58f21c-istiod-ca-cert\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:00.956746 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.956425    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-istio-data\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:00.956746 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.956477    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f891215-2cc6-4b56-8ba2-68687e58f21c-istio-podinfo\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:00.956746 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.956553    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-workload-socket\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:00.956746 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.956616    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-credential-socket\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:00.956746 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.956649    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-workload-certs\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:00.961388 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.959997    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f891215-2cc6-4b56-8ba2-68687e58f21c-istiod-ca-cert\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:00.961388 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.960252    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-workload-socket\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:00.961388 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.960517    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-credential-socket\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:00.961388 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.960746    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-istio-data\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:00.967532 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.957340    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-workload-certs\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:00.967940 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.967879    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-istio-envoy\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:00.969751 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:00.969730    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f891215-2cc6-4b56-8ba2-68687e58f21c-istio-podinfo\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:01.015768 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:01.015732    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zlwm\" (UniqueName: \"kubernetes.io/projected/6f891215-2cc6-4b56-8ba2-68687e58f21c-kube-api-access-8zlwm\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:01.015768 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:01.015749    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f891215-2cc6-4b56-8ba2-68687e58f21c-istio-token\") pod \"e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") " pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:01.064218 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:01.064185    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:01.367817 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:01.367772    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5" event={"ID":"9db8cae8-a410-4c25-bdec-cee65fb1a8ae","Type":"ContainerStarted","Data":"c8d1e942eeb3d42c6466daabdb66e4ffb8bc859c036ba5b9caf71d08ea040b3f"}
Oct 30 21:56:01.368278 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:01.367829    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5" event={"ID":"9db8cae8-a410-4c25-bdec-cee65fb1a8ae","Type":"ContainerStarted","Data":"91949efa101975e9eaef2bcaf421d5847138e0ba841e2aef3a5f91bb4daf2d45"}
Oct 30 21:56:01.370122 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:01.370089    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96" event={"ID":"727b252d-ad3e-4e43-9fb8-471fd2957231","Type":"ContainerStarted","Data":"2a184e22927900b74b6226bf17a18ebf3d0c92e1216ff49aad0d86bc8c54e247"}
Oct 30 21:56:01.370280 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:01.370147    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96" event={"ID":"727b252d-ad3e-4e43-9fb8-471fd2957231","Type":"ContainerStarted","Data":"79367c6f291c911af631d88b7ea6c6f99c0bf9c0aa7b427374b528248873d2f4"}
Oct 30 21:56:01.372896 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:01.372870    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6afc4d44-gdqgr" event={"ID":"30ad2bc8-ec58-4b55-8c49-869450d196f6","Type":"ContainerStarted","Data":"685bcae5ac5a6d78af32ed17e8242a111f72b18f6df200f3b955a9c7683f6929"}
Oct 30 21:56:01.372961 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:01.372925    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6afc4d44-gdqgr" event={"ID":"30ad2bc8-ec58-4b55-8c49-869450d196f6","Type":"ContainerStarted","Data":"4ba76ab3bf0e17cccdd6bce1e32944d81cdcc7c614d321f4c61b6b7f6132e0e1"}
Oct 30 21:56:01.418421 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:01.418389    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:01.420244 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:01.420216    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:56:01.420402 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:01.420265    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96" podUID="727b252d-ad3e-4e43-9fb8-471fd2957231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 30 21:56:01.491036 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:01.491007    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"]
Oct 30 21:56:01.520833 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:01.520797    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:56:01.520907 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:01.520867    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:56:01.520907 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:01.520898    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:56:01.779919 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:01.779851    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6afc4d44-gdqgr" podStartSLOduration=2.779830301 podStartE2EDuration="2.779830301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:01.765283182 +0000 UTC m=+1356.239787161" watchObservedRunningTime="2026-10-30 21:56:01.779830301 +0000 UTC m=+1356.254334281"
Oct 30 21:56:01.781212 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:01.781155    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5" podStartSLOduration=2.781139129 podStartE2EDuration="2.781139129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:01.58465092 +0000 UTC m=+1356.059154900" watchObservedRunningTime="2026-10-30 21:56:01.781139129 +0000 UTC m=+1356.255643110"
Oct 30 21:56:01.857373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:01.857265    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:56:01.858494 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:01.858463    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:56:01.858723 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:01.858697    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5" podUID="9db8cae8-a410-4c25-bdec-cee65fb1a8ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 21:56:01.905663 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:01.905552    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96" podStartSLOduration=1.905531606 podStartE2EDuration="1.905531606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:01.903735359 +0000 UTC m=+1356.378239335" watchObservedRunningTime="2026-10-30 21:56:01.905531606 +0000 UTC m=+1356.380035585"
Oct 30 21:56:02.409221 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:02.409172    5087 generic.go:356] "Generic (PLEG): container finished" podID="30ad2bc8-ec58-4b55-8c49-869450d196f6" containerID="685bcae5ac5a6d78af32ed17e8242a111f72b18f6df200f3b955a9c7683f6929" exitCode=0
Oct 30 21:56:02.410153 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:02.410124    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6afc4d44-gdqgr" event={"ID":"30ad2bc8-ec58-4b55-8c49-869450d196f6","Type":"ContainerDied","Data":"685bcae5ac5a6d78af32ed17e8242a111f72b18f6df200f3b955a9c7683f6929"}
Oct 30 21:56:02.415505 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:02.415471    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk" event={"ID":"6f891215-2cc6-4b56-8ba2-68687e58f21c","Type":"ContainerStarted","Data":"d5bc6fca893b137116b81e4aa445275f19416b0b9faad4743e2ca6d3b960bebb"}
Oct 30 21:56:02.416312 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:02.416270    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk" event={"ID":"6f891215-2cc6-4b56-8ba2-68687e58f21c","Type":"ContainerStarted","Data":"910a74cea5d2d0934daf4ae55946b23e184c8d84ff330aa16d4df4e31c2d02ed"}
Oct 30 21:56:02.430391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:02.428497    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:56:02.430391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:02.428574    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96" podUID="727b252d-ad3e-4e43-9fb8-471fd2957231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 30 21:56:02.867374 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:02.863514    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:56:02.867374 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:02.863573    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5" podUID="9db8cae8-a410-4c25-bdec-cee65fb1a8ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 21:56:03.084372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:03.071429    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:03.084372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:03.071555    5087 patch_prober.go:29] interesting pod/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:56:03.084372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:03.071602    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk" podUID="6f891215-2cc6-4b56-8ba2-68687e58f21c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 21:56:03.434386 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:03.433830    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:56:03.434386 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:03.433903    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96" podUID="727b252d-ad3e-4e43-9fb8-471fd2957231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 30 21:56:03.871374 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:03.867603    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:03.871374 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:03.868621    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:56:03.871374 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:03.868661    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5" podUID="9db8cae8-a410-4c25-bdec-cee65fb1a8ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 21:56:04.088317 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:04.088283    5087 patch_prober.go:29] interesting pod/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:56:04.088669 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:04.088620    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk" podUID="6f891215-2cc6-4b56-8ba2-68687e58f21c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 21:56:04.125378 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:04.119340    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk" podStartSLOduration=4.119317842 podStartE2EDuration="4.119317842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:02.711131183 +0000 UTC m=+1357.185635163" watchObservedRunningTime="2026-10-30 21:56:04.119317842 +0000 UTC m=+1358.593821819"
Oct 30 21:56:04.164625 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:04.146792    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3ca8f96-j62t6"]
Oct 30 21:56:04.164625 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:04.146860    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3ca8f96-j62t6"]
Oct 30 21:56:04.418826 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:04.418791    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:56:04.419091 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:04.419061    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96" podUID="727b252d-ad3e-4e43-9fb8-471fd2957231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 30 21:56:04.430886 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:04.430864    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6afc4d44-gdqgr"
Oct 30 21:56:04.527113 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:04.527079    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30ad2bc8-ec58-4b55-8c49-869450d196f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ad2bc8-ec58-4b55-8c49-869450d196f6-maas-api-openshift-ca\") pod \"30ad2bc8-ec58-4b55-8c49-869450d196f6\" (UID: \"30ad2bc8-ec58-4b55-8c49-869450d196f6\") "
Oct 30 21:56:04.527511 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:04.527253    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30ad2bc8-ec58-4b55-8c49-869450d196f6-kube-api-access-kdv69\" (UniqueName: \"kubernetes.io/projected/30ad2bc8-ec58-4b55-8c49-869450d196f6-kube-api-access-kdv69\") pod \"30ad2bc8-ec58-4b55-8c49-869450d196f6\" (UID: \"30ad2bc8-ec58-4b55-8c49-869450d196f6\") "
Oct 30 21:56:04.527511 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:04.527378    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30ad2bc8-ec58-4b55-8c49-869450d196f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ad2bc8-ec58-4b55-8c49-869450d196f6-maas-api-xks-ca\") pod \"30ad2bc8-ec58-4b55-8c49-869450d196f6\" (UID: \"30ad2bc8-ec58-4b55-8c49-869450d196f6\") "
Oct 30 21:56:04.527655 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:04.527553    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ad2bc8-ec58-4b55-8c49-869450d196f6-maas-api-openshift-ca" pod "30ad2bc8-ec58-4b55-8c49-869450d196f6" (UID: "30ad2bc8-ec58-4b55-8c49-869450d196f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:04.527906 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:04.527861    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ad2bc8-ec58-4b55-8c49-869450d196f6-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:56:04.529399 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:04.529371    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30ad2bc8-ec58-4b55-8c49-869450d196f6-maas-api-xks-ca" pod "30ad2bc8-ec58-4b55-8c49-869450d196f6" (UID: "30ad2bc8-ec58-4b55-8c49-869450d196f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:04.529710 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:04.529685    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ad2bc8-ec58-4b55-8c49-869450d196f6-kube-api-access-kdv69" pod "30ad2bc8-ec58-4b55-8c49-869450d196f6" (UID: "30ad2bc8-ec58-4b55-8c49-869450d196f6"). InnerVolumeSpecName "kube-api-access-kdv69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:04.658494 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:04.652283    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdv69\" (UniqueName: \"kubernetes.io/projected/30ad2bc8-ec58-4b55-8c49-869450d196f6-kube-api-access-kdv69\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:56:04.658494 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:04.652327    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ad2bc8-ec58-4b55-8c49-869450d196f6-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:56:04.831451 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:04.817553    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6afc4d44-gdqgr"
Oct 30 21:56:04.831451 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:04.828546    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6afc4d44-gdqgr" event={"ID":"30ad2bc8-ec58-4b55-8c49-869450d196f6","Type":"ContainerDied","Data":"4ba76ab3bf0e17cccdd6bce1e32944d81cdcc7c614d321f4c61b6b7f6132e0e1"}
Oct 30 21:56:04.831451 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:04.828599    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ba76ab3bf0e17cccdd6bce1e32944d81cdcc7c614d321f4c61b6b7f6132e0e1"
Oct 30 21:56:04.876588 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:04.858573    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:56:04.876588 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:04.858624    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5" podUID="9db8cae8-a410-4c25-bdec-cee65fb1a8ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 21:56:05.069373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:05.068557    5087 patch_prober.go:29] interesting pod/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:56:05.069373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:05.068649    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk" podUID="6f891215-2cc6-4b56-8ba2-68687e58f21c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 21:56:05.377705 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:05.376917    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d727205-75vqz"]
Oct 30 21:56:05.381385 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:05.377893    5087 cpu_manager.go:414] "containerMap: removing container" podUID="30ad2bc8-ec58-4b55-8c49-869450d196f6" containerName="revoke-keys"
Oct 30 21:56:05.381385 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:05.377917    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30ad2bc8-ec58-4b55-8c49-869450d196f6" containerName="revoke-keys"
Oct 30 21:56:05.381385 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:05.378078    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="30ad2bc8-ec58-4b55-8c49-869450d196f6" containerName="revoke-keys"
Oct 30 21:56:05.432375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:05.431779    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:56:05.432375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:05.431835    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96" podUID="727b252d-ad3e-4e43-9fb8-471fd2957231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 30 21:56:05.435424 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:05.433601    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d727205-75vqz"
Oct 30 21:56:05.450374 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:05.436964    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d727205-75vqz"]
Oct 30 21:56:05.525401 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:05.490778    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:56:05.525401 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:05.501317    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:56:05.618186 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:05.618150    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3379ae7c-a890-424f-9b75-fae49bd41850-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d727205-75vqz\" (UID: \"3379ae7c-a890-424f-9b75-fae49bd41850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d727205-75vqz"
Oct 30 21:56:05.618877 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:05.618843    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3379ae7c-a890-424f-9b75-fae49bd41850-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d727205-75vqz\" (UID: \"3379ae7c-a890-424f-9b75-fae49bd41850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d727205-75vqz"
Oct 30 21:56:05.619186 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:05.619166    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krk5b\" (UniqueName: \"kubernetes.io/projected/3379ae7c-a890-424f-9b75-fae49bd41850-kube-api-access-krk5b\") pod \"maas-api-revoke-sub-6d727205-75vqz\" (UID: \"3379ae7c-a890-424f-9b75-fae49bd41850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d727205-75vqz"
Oct 30 21:56:05.721518 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:05.721071    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krk5b\" (UniqueName: \"kubernetes.io/projected/3379ae7c-a890-424f-9b75-fae49bd41850-kube-api-access-krk5b\") pod \"maas-api-revoke-sub-6d727205-75vqz\" (UID: \"3379ae7c-a890-424f-9b75-fae49bd41850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d727205-75vqz"
Oct 30 21:56:05.721518 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:05.721176    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3379ae7c-a890-424f-9b75-fae49bd41850-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d727205-75vqz\" (UID: \"3379ae7c-a890-424f-9b75-fae49bd41850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d727205-75vqz"
Oct 30 21:56:05.721518 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:05.721289    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3379ae7c-a890-424f-9b75-fae49bd41850-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d727205-75vqz\" (UID: \"3379ae7c-a890-424f-9b75-fae49bd41850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d727205-75vqz"
Oct 30 21:56:05.722337 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:05.722310    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3379ae7c-a890-424f-9b75-fae49bd41850-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d727205-75vqz\" (UID: \"3379ae7c-a890-424f-9b75-fae49bd41850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d727205-75vqz"
Oct 30 21:56:05.723811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:05.723781    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3379ae7c-a890-424f-9b75-fae49bd41850-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d727205-75vqz\" (UID: \"3379ae7c-a890-424f-9b75-fae49bd41850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d727205-75vqz"
Oct 30 21:56:05.729533 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:05.729508    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfee1062-17e6-4d74-9140-f4eed1f21a07" path="/var/lib/kubelet/pods/bfee1062-17e6-4d74-9140-f4eed1f21a07/volumes"
Oct 30 21:56:05.795374 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:05.795253    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krk5b\" (UniqueName: \"kubernetes.io/projected/3379ae7c-a890-424f-9b75-fae49bd41850-kube-api-access-krk5b\") pod \"maas-api-revoke-sub-6d727205-75vqz\" (UID: \"3379ae7c-a890-424f-9b75-fae49bd41850\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d727205-75vqz"
Oct 30 21:56:05.849735 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:05.846951    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d727205-75vqz"
Oct 30 21:56:05.936500 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:05.931516    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:56:05.936500 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:05.931578    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5" podUID="9db8cae8-a410-4c25-bdec-cee65fb1a8ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 21:56:06.096375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:06.076644    5087 patch_prober.go:29] interesting pod/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:56:06.096375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:06.076702    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk" podUID="6f891215-2cc6-4b56-8ba2-68687e58f21c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 21:56:06.449488 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:06.449448    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:56:06.449766 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:06.449734    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96" podUID="727b252d-ad3e-4e43-9fb8-471fd2957231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 30 21:56:06.765875 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:06.765832    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d727205-75vqz"]
Oct 30 21:56:06.804392 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:06.802857    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:06.839790 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:06.839433    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:06.906382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:06.901840    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:56:06.906382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:06.901919    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5" podUID="9db8cae8-a410-4c25-bdec-cee65fb1a8ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 21:56:06.916955 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:06.916887    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d727205-75vqz" event={"ID":"3379ae7c-a890-424f-9b75-fae49bd41850","Type":"ContainerStarted","Data":"9d698f6e6813f71664e2af86951e696e02a87b11207c70978061e94f930c0804"}
Oct 30 21:56:06.973126 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:06.967477    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:07.024369 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:07.024197    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:07.064717 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:07.064681    5087 patch_prober.go:29] interesting pod/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:56:07.065001 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:07.064973    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk" podUID="6f891215-2cc6-4b56-8ba2-68687e58f21c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 21:56:07.418369 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:07.418249    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 30 21:56:07.418369 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:07.418301    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96" podUID="727b252d-ad3e-4e43-9fb8-471fd2957231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 30 21:56:07.859372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:07.856639    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Oct 30 21:56:07.859372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:07.856698    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5" podUID="9db8cae8-a410-4c25-bdec-cee65fb1a8ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Oct 30 21:56:07.931365 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:07.931182    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d727205-75vqz" event={"ID":"3379ae7c-a890-424f-9b75-fae49bd41850","Type":"ContainerStarted","Data":"786f538dc8fb4ec0bac9f57a072c52b8b517752d4f3a631ae388b9ca1ace3b9a"}
Oct 30 21:56:08.077375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:08.063769    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d727205-75vqz" podStartSLOduration=3.063747857 podStartE2EDuration="3.063747857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:08.058761836 +0000 UTC m=+1362.533265828" watchObservedRunningTime="2026-10-30 21:56:08.063747857 +0000 UTC m=+1362.538251837"
Oct 30 21:56:08.088384 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:08.087131    5087 patch_prober.go:29] interesting pod/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:56:08.088384 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:08.087184    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk" podUID="6f891215-2cc6-4b56-8ba2-68687e58f21c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 21:56:08.104544 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:08.104508    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38fb985e-96797"]
Oct 30 21:56:08.116010 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:08.115986    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38fb985e-96797"]
Oct 30 21:56:08.878176 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:08.878143    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:56:08.880275 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:08.880127    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:56:08.881231 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:08.881190    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 21:56:08.900180 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:08.899555    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:08.900377 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:08.900211    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:08.901330 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:08.901010    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 21:56:08.944620 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:08.944577    5087 generic.go:356] "Generic (PLEG): container finished" podID="3379ae7c-a890-424f-9b75-fae49bd41850" containerID="786f538dc8fb4ec0bac9f57a072c52b8b517752d4f3a631ae388b9ca1ace3b9a" exitCode=0
Oct 30 21:56:08.946441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:08.945670    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d727205-75vqz" event={"ID":"3379ae7c-a890-424f-9b75-fae49bd41850","Type":"ContainerDied","Data":"786f538dc8fb4ec0bac9f57a072c52b8b517752d4f3a631ae388b9ca1ace3b9a"}
Oct 30 21:56:09.065225 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:09.065190    5087 patch_prober.go:29] interesting pod/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body=
Oct 30 21:56:09.065417 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:09.065255    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk" podUID="6f891215-2cc6-4b56-8ba2-68687e58f21c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused"
Oct 30 21:56:09.712770 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:09.712738    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e3bcf59-d6bb-4198-a397-8d4432360425" path="/var/lib/kubelet/pods/9e3bcf59-d6bb-4198-a397-8d4432360425/volumes"
Oct 30 21:56:10.082922 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:10.081883    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:10.082922 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:10.082816    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:10.084082 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:10.084038    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 21:56:10.136186 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:10.136156    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d727205-75vqz"
Oct 30 21:56:10.206564 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:10.206534    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3379ae7c-a890-424f-9b75-fae49bd41850-kube-api-access-krk5b\" (UniqueName: \"kubernetes.io/projected/3379ae7c-a890-424f-9b75-fae49bd41850-kube-api-access-krk5b\") pod \"3379ae7c-a890-424f-9b75-fae49bd41850\" (UID: \"3379ae7c-a890-424f-9b75-fae49bd41850\") "
Oct 30 21:56:10.206731 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:10.206570    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3379ae7c-a890-424f-9b75-fae49bd41850-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3379ae7c-a890-424f-9b75-fae49bd41850-maas-api-xks-ca\") pod \"3379ae7c-a890-424f-9b75-fae49bd41850\" (UID: \"3379ae7c-a890-424f-9b75-fae49bd41850\") "
Oct 30 21:56:10.206790 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:10.206771    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3379ae7c-a890-424f-9b75-fae49bd41850-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3379ae7c-a890-424f-9b75-fae49bd41850-maas-api-openshift-ca\") pod \"3379ae7c-a890-424f-9b75-fae49bd41850\" (UID: \"3379ae7c-a890-424f-9b75-fae49bd41850\") "
Oct 30 21:56:10.207092 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:10.207063    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3379ae7c-a890-424f-9b75-fae49bd41850-maas-api-openshift-ca" pod "3379ae7c-a890-424f-9b75-fae49bd41850" (UID: "3379ae7c-a890-424f-9b75-fae49bd41850"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:10.207367 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:10.207331    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3379ae7c-a890-424f-9b75-fae49bd41850-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:56:10.208784 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:10.208765    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3379ae7c-a890-424f-9b75-fae49bd41850-maas-api-xks-ca" pod "3379ae7c-a890-424f-9b75-fae49bd41850" (UID: "3379ae7c-a890-424f-9b75-fae49bd41850"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:10.208843 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:10.208813    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3379ae7c-a890-424f-9b75-fae49bd41850-kube-api-access-krk5b" pod "3379ae7c-a890-424f-9b75-fae49bd41850" (UID: "3379ae7c-a890-424f-9b75-fae49bd41850"). InnerVolumeSpecName "kube-api-access-krk5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:10.308797 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:10.308710    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3379ae7c-a890-424f-9b75-fae49bd41850-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:56:10.308797 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:10.308739    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krk5b\" (UniqueName: \"kubernetes.io/projected/3379ae7c-a890-424f-9b75-fae49bd41850-kube-api-access-krk5b\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:56:10.970403 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:10.967631    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d727205-75vqz"
Oct 30 21:56:10.970403 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:10.967818    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d727205-75vqz" event={"ID":"3379ae7c-a890-424f-9b75-fae49bd41850","Type":"ContainerDied","Data":"9d698f6e6813f71664e2af86951e696e02a87b11207c70978061e94f930c0804"}
Oct 30 21:56:10.970403 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:10.967866    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d698f6e6813f71664e2af86951e696e02a87b11207c70978061e94f930c0804"
Oct 30 21:56:11.094265 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:11.094231    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41cf4770-bq55n"]
Oct 30 21:56:11.098182 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:11.098150    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41cf4770-bq55n"]
Oct 30 21:56:11.710175 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:11.710145    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4640ac0e-2df6-42ca-8ff9-8b32d1e386fa" path="/var/lib/kubelet/pods/4640ac0e-2df6-42ca-8ff9-8b32d1e386fa/volumes"
Oct 30 21:56:12.314068 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:12.314034    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-139b6711-vp8hl"]
Oct 30 21:56:12.314646 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:12.314633    5087 cpu_manager.go:414] "containerMap: removing container" podUID="3379ae7c-a890-424f-9b75-fae49bd41850" containerName="revoke-keys"
Oct 30 21:56:12.314692 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:12.314647    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3379ae7c-a890-424f-9b75-fae49bd41850" containerName="revoke-keys"
Oct 30 21:56:12.314762 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:12.314753    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="3379ae7c-a890-424f-9b75-fae49bd41850" containerName="revoke-keys"
Oct 30 21:56:12.318385 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:12.318366    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-139b6711-vp8hl"
Oct 30 21:56:12.330625 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:12.330603    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b644d863-029e-4159-9259-0e72a2b021fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-139b6711-vp8hl\" (UID: \"b644d863-029e-4159-9259-0e72a2b021fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-139b6711-vp8hl"
Oct 30 21:56:12.330720 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:12.330653    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b644d863-029e-4159-9259-0e72a2b021fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-139b6711-vp8hl\" (UID: \"b644d863-029e-4159-9259-0e72a2b021fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-139b6711-vp8hl"
Oct 30 21:56:12.330762 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:12.330730    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr8nw\" (UniqueName: \"kubernetes.io/projected/b644d863-029e-4159-9259-0e72a2b021fd-kube-api-access-nr8nw\") pod \"maas-api-revoke-sub-139b6711-vp8hl\" (UID: \"b644d863-029e-4159-9259-0e72a2b021fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-139b6711-vp8hl"
Oct 30 21:56:12.426558 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:12.426530    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:56:12.426761 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:12.426537    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:56:12.429295 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:12.429274    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-139b6711-vp8hl"]
Oct 30 21:56:12.432104 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:12.432083    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b644d863-029e-4159-9259-0e72a2b021fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-139b6711-vp8hl\" (UID: \"b644d863-029e-4159-9259-0e72a2b021fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-139b6711-vp8hl"
Oct 30 21:56:12.432212 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:12.432126    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr8nw\" (UniqueName: \"kubernetes.io/projected/b644d863-029e-4159-9259-0e72a2b021fd-kube-api-access-nr8nw\") pod \"maas-api-revoke-sub-139b6711-vp8hl\" (UID: \"b644d863-029e-4159-9259-0e72a2b021fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-139b6711-vp8hl"
Oct 30 21:56:12.432361 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:12.432324    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b644d863-029e-4159-9259-0e72a2b021fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-139b6711-vp8hl\" (UID: \"b644d863-029e-4159-9259-0e72a2b021fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-139b6711-vp8hl"
Oct 30 21:56:12.432927 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:12.432907    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b644d863-029e-4159-9259-0e72a2b021fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-139b6711-vp8hl\" (UID: \"b644d863-029e-4159-9259-0e72a2b021fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-139b6711-vp8hl"
Oct 30 21:56:12.434609 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:12.434591    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b644d863-029e-4159-9259-0e72a2b021fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-139b6711-vp8hl\" (UID: \"b644d863-029e-4159-9259-0e72a2b021fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-139b6711-vp8hl"
Oct 30 21:56:12.489682 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:12.489657    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr8nw\" (UniqueName: \"kubernetes.io/projected/b644d863-029e-4159-9259-0e72a2b021fd-kube-api-access-nr8nw\") pod \"maas-api-revoke-sub-139b6711-vp8hl\" (UID: \"b644d863-029e-4159-9259-0e72a2b021fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-139b6711-vp8hl"
Oct 30 21:56:12.632437 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:12.632363    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-139b6711-vp8hl"
Oct 30 21:56:13.023887 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:13.023859    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-139b6711-vp8hl"]
Oct 30 21:56:13.024232 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:56:13.024207    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb644d863_029e_4159_9259_0e72a2b021fd.slice/crio-d80842628659fd8198fbb20fcb8167fd259f351ab0dd2e9e0a66bca57c04dc44 WatchSource:0}: Error finding container d80842628659fd8198fbb20fcb8167fd259f351ab0dd2e9e0a66bca57c04dc44: Status 404 returned error can't find the container with id d80842628659fd8198fbb20fcb8167fd259f351ab0dd2e9e0a66bca57c04dc44
Oct 30 21:56:13.993827 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:13.993789    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-139b6711-vp8hl" event={"ID":"b644d863-029e-4159-9259-0e72a2b021fd","Type":"ContainerStarted","Data":"ef8f6b66b20ae7932faff32d0688d08b4eeda2c08d7ec410b0b9cca260c78f86"}
Oct 30 21:56:13.994212 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:13.993844    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-139b6711-vp8hl" event={"ID":"b644d863-029e-4159-9259-0e72a2b021fd","Type":"ContainerStarted","Data":"d80842628659fd8198fbb20fcb8167fd259f351ab0dd2e9e0a66bca57c04dc44"}
Oct 30 21:56:14.112508 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:14.112449    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-139b6711-vp8hl" podStartSLOduration=2.112433889 podStartE2EDuration="2.112433889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:14.109085467 +0000 UTC m=+1368.583589534" watchObservedRunningTime="2026-10-30 21:56:14.112433889 +0000 UTC m=+1368.586937868"
Oct 30 21:56:15.022371 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:15.021732    5087 generic.go:356] "Generic (PLEG): container finished" podID="b644d863-029e-4159-9259-0e72a2b021fd" containerID="ef8f6b66b20ae7932faff32d0688d08b4eeda2c08d7ec410b0b9cca260c78f86" exitCode=0
Oct 30 21:56:15.022371 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:15.021835    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-139b6711-vp8hl" event={"ID":"b644d863-029e-4159-9259-0e72a2b021fd","Type":"ContainerDied","Data":"ef8f6b66b20ae7932faff32d0688d08b4eeda2c08d7ec410b0b9cca260c78f86"}
Oct 30 21:56:16.065073 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:16.065046    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18b65dbd-bxxqx"]
Oct 30 21:56:16.161392 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:16.161334    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18b65dbd-bxxqx"]
Oct 30 21:56:16.972699 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:16.972605    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:16.973453 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:16.973418    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 21:56:16.986663 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:16.986623    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:17.082545 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:16.995668    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:56:17.082545 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:16.997380    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:17.082545 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:17.024705    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:17.596735 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:17.587621    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:56:17.643191 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:17.643155    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-139b6711-vp8hl"
Oct 30 21:56:17.755375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:17.751810    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b644d863-029e-4159-9259-0e72a2b021fd-kube-api-access-nr8nw\" (UniqueName: \"kubernetes.io/projected/b644d863-029e-4159-9259-0e72a2b021fd-kube-api-access-nr8nw\") pod \"b644d863-029e-4159-9259-0e72a2b021fd\" (UID: \"b644d863-029e-4159-9259-0e72a2b021fd\") "
Oct 30 21:56:17.755375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:17.751977    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b644d863-029e-4159-9259-0e72a2b021fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b644d863-029e-4159-9259-0e72a2b021fd-maas-api-openshift-ca\") pod \"b644d863-029e-4159-9259-0e72a2b021fd\" (UID: \"b644d863-029e-4159-9259-0e72a2b021fd\") "
Oct 30 21:56:17.755375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:17.752051    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b644d863-029e-4159-9259-0e72a2b021fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b644d863-029e-4159-9259-0e72a2b021fd-maas-api-xks-ca\") pod \"b644d863-029e-4159-9259-0e72a2b021fd\" (UID: \"b644d863-029e-4159-9259-0e72a2b021fd\") "
Oct 30 21:56:17.781617 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:17.781575    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b644d863-029e-4159-9259-0e72a2b021fd-maas-api-openshift-ca" pod "b644d863-029e-4159-9259-0e72a2b021fd" (UID: "b644d863-029e-4159-9259-0e72a2b021fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:17.811590 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:17.811553    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b644d863-029e-4159-9259-0e72a2b021fd-maas-api-xks-ca" pod "b644d863-029e-4159-9259-0e72a2b021fd" (UID: "b644d863-029e-4159-9259-0e72a2b021fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:17.838139 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:17.838102    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b644d863-029e-4159-9259-0e72a2b021fd-kube-api-access-nr8nw" pod "b644d863-029e-4159-9259-0e72a2b021fd" (UID: "b644d863-029e-4159-9259-0e72a2b021fd"). InnerVolumeSpecName "kube-api-access-nr8nw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:17.852337 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:56:17.852082    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:56:17.865590 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:17.864175    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7171f5cd-a37b-499f-a06a-2a9b4a121da2" path="/var/lib/kubelet/pods/7171f5cd-a37b-499f-a06a-2a9b4a121da2/volumes"
Oct 30 21:56:17.951439 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:17.905163    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b644d863-029e-4159-9259-0e72a2b021fd-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:56:17.951747 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:17.951717    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b644d863-029e-4159-9259-0e72a2b021fd-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:56:17.964374 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:17.951857    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr8nw\" (UniqueName: \"kubernetes.io/projected/b644d863-029e-4159-9259-0e72a2b021fd-kube-api-access-nr8nw\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:56:17.989488 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:17.989443    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:18.004403 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:17.990387    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:56:18.049437 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:18.033320    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:18.065599 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:18.065568    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:56:18.066665 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:18.045315    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:18.066953 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:18.045257    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:18.081371 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:18.045337    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 21:56:18.081860 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:18.045769    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:56:18.082195 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:18.045789    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 21:56:18.082532 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:18.045810    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:56:18.082853 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:18.045854    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:56:18.083505 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:18.045874    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Oct 30 21:56:18.083816 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:18.045739    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml" podUID="01fdf382-6833-463a-bed9-c7c096166819" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:18.084102 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:18.068077    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:56:18.084261 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:18.084247    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:56:18.084391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:18.084381    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:56:18.084534 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:18.084524    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:56:18.106373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:18.087300    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc" podUID="f6a9f73b-bb1f-4270-a38d-3703c46fae07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:18.358431 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:18.358246    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:18.358431 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:18.358370    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml" podUID="01fdf382-6833-463a-bed9-c7c096166819" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": context deadline exceeded"
Oct 30 21:56:18.760442 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:18.745686    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:56:18.760442 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:18.746744    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:56:18.803375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:18.800118    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:18.878379 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:18.872300    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-139b6711-vp8hl" event={"ID":"b644d863-029e-4159-9259-0e72a2b021fd","Type":"ContainerDied","Data":"d80842628659fd8198fbb20fcb8167fd259f351ab0dd2e9e0a66bca57c04dc44"}
Oct 30 21:56:18.879174 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:18.879141    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d80842628659fd8198fbb20fcb8167fd259f351ab0dd2e9e0a66bca57c04dc44"
Oct 30 21:56:18.879339 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:18.878913    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-139b6711-vp8hl"
Oct 30 21:56:19.085292 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:19.085173    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml" podUID="01fdf382-6833-463a-bed9-c7c096166819" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:19.086364 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:19.086317    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:56:19.086878 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:19.086853    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 21:56:19.087478 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:19.087314    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:56:19.088427 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:19.088402    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 21:56:19.090941 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:19.090918    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:56:19.091341 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:19.091319    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:56:19.091727 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:19.091705    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Oct 30 21:56:19.092118 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:19.092098    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:56:19.092281 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:19.092268    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:56:19.092433 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:19.092422    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:56:19.092552 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:19.092543    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:56:19.092656 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:19.092646    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:56:19.102617 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:19.102588    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:56:19.103523 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:19.103493    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:19.104072 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:19.104048    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": context deadline exceeded"
Oct 30 21:56:19.104286 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:19.104265    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:56:19.136376 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:19.134792    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc" podUID="f6a9f73b-bb1f-4270-a38d-3703c46fae07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:19.136376 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:19.135146    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:56:19.169370 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:19.167473    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 30 21:56:19.422116 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:19.422029    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:56:19.578372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:19.578112    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e360879d-txmm9"]
Oct 30 21:56:19.682512 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:19.682407    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e360879d-txmm9"]
Oct 30 21:56:19.754230 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:19.754157    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6d4959a-1688-4683-be56-a7e208105df2" path="/var/lib/kubelet/pods/d6d4959a-1688-4683-be56-a7e208105df2/volumes"
Oct 30 21:56:19.795473 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:19.795433    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:19.795741 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:19.795715    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": context deadline exceeded"
Oct 30 21:56:19.904823 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:19.902551    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:56:19.989377 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:19.989068    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:56:20.090373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:20.086946    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml" podUID="01fdf382-6833-463a-bed9-c7c096166819" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:20.090373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:20.087374    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:20.109412 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:20.093420    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:20.109412 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:20.108957    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:56:20.125422 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:20.124164    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:56:20.126698 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:20.125665    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:20.139690 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:20.134824    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 21:56:20.139690 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:20.137027    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc" podUID="f6a9f73b-bb1f-4270-a38d-3703c46fae07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:20.139690 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:20.137232    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:56:20.170654 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:20.170468    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:20.261920 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:20.257618    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:56:20.274413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:20.262728    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:56:20.304643 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:20.304614    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:56:20.429676 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:20.429621    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:21.030553 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:21.029834    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:56:21.126568 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:21.126451    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:21.128110 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:21.128074    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:21.146015 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:21.137793    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:21.146015 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:21.138331    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc" podUID="f6a9f73b-bb1f-4270-a38d-3703c46fae07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:21.157835 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:21.157798    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Oct 30 21:56:21.159563 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:21.159537    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:56:21.178464 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:21.178443    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:56:21.205072 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:21.205049    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:56:21.214410 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:21.212091    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:56:21.238368 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:21.237806    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:56:21.245685 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:21.245652    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:56:21.582412 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:21.582378    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e31d16af-2r9pz"]
Oct 30 21:56:21.583044 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:21.583024    5087 cpu_manager.go:414] "containerMap: removing container" podUID="b644d863-029e-4159-9259-0e72a2b021fd" containerName="revoke-keys"
Oct 30 21:56:21.583145 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:21.583045    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b644d863-029e-4159-9259-0e72a2b021fd" containerName="revoke-keys"
Oct 30 21:56:21.583249 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:21.583236    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="b644d863-029e-4159-9259-0e72a2b021fd" containerName="revoke-keys"
Oct 30 21:56:21.588227 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:21.588207    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31d16af-2r9pz"
Oct 30 21:56:21.627052 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:21.627020    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9cvc\" (UniqueName: \"kubernetes.io/projected/3994e998-c96e-4c4f-8c5d-6f7f7ce895a8-kube-api-access-w9cvc\") pod \"maas-api-revoke-sub-e31d16af-2r9pz\" (UID: \"3994e998-c96e-4c4f-8c5d-6f7f7ce895a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31d16af-2r9pz"
Oct 30 21:56:21.627173 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:21.627158    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3994e998-c96e-4c4f-8c5d-6f7f7ce895a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e31d16af-2r9pz\" (UID: \"3994e998-c96e-4c4f-8c5d-6f7f7ce895a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31d16af-2r9pz"
Oct 30 21:56:21.627218 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:21.627204    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3994e998-c96e-4c4f-8c5d-6f7f7ce895a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e31d16af-2r9pz\" (UID: \"3994e998-c96e-4c4f-8c5d-6f7f7ce895a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31d16af-2r9pz"
Oct 30 21:56:21.642440 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:21.642417    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:56:21.642615 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:21.642441    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:56:21.650611 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:21.650581    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e31d16af-2r9pz"]
Oct 30 21:56:21.728677 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:21.728643    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9cvc\" (UniqueName: \"kubernetes.io/projected/3994e998-c96e-4c4f-8c5d-6f7f7ce895a8-kube-api-access-w9cvc\") pod \"maas-api-revoke-sub-e31d16af-2r9pz\" (UID: \"3994e998-c96e-4c4f-8c5d-6f7f7ce895a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31d16af-2r9pz"
Oct 30 21:56:21.728831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:21.728731    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3994e998-c96e-4c4f-8c5d-6f7f7ce895a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e31d16af-2r9pz\" (UID: \"3994e998-c96e-4c4f-8c5d-6f7f7ce895a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31d16af-2r9pz"
Oct 30 21:56:21.728831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:21.728773    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3994e998-c96e-4c4f-8c5d-6f7f7ce895a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e31d16af-2r9pz\" (UID: \"3994e998-c96e-4c4f-8c5d-6f7f7ce895a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31d16af-2r9pz"
Oct 30 21:56:21.729671 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:21.729644    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3994e998-c96e-4c4f-8c5d-6f7f7ce895a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e31d16af-2r9pz\" (UID: \"3994e998-c96e-4c4f-8c5d-6f7f7ce895a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31d16af-2r9pz"
Oct 30 21:56:21.731505 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:21.731472    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3994e998-c96e-4c4f-8c5d-6f7f7ce895a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e31d16af-2r9pz\" (UID: \"3994e998-c96e-4c4f-8c5d-6f7f7ce895a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31d16af-2r9pz"
Oct 30 21:56:21.786642 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:21.786604    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9cvc\" (UniqueName: \"kubernetes.io/projected/3994e998-c96e-4c4f-8c5d-6f7f7ce895a8-kube-api-access-w9cvc\") pod \"maas-api-revoke-sub-e31d16af-2r9pz\" (UID: \"3994e998-c96e-4c4f-8c5d-6f7f7ce895a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31d16af-2r9pz"
Oct 30 21:56:21.902912 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:21.902822    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31d16af-2r9pz"
Oct 30 21:56:22.307506 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:22.302999    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e31d16af-2r9pz"]
Oct 30 21:56:22.308087 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:56:22.307619    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3994e998_c96e_4c4f_8c5d_6f7f7ce895a8.slice/crio-738028718a7f99e45cc2c2caf2e1bfcda264205449f6ad9b6771fe5fe6f8eb7f WatchSource:0}: Error finding container 738028718a7f99e45cc2c2caf2e1bfcda264205449f6ad9b6771fe5fe6f8eb7f: Status 404 returned error can't find the container with id 738028718a7f99e45cc2c2caf2e1bfcda264205449f6ad9b6771fe5fe6f8eb7f
Oct 30 21:56:23.275790 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:23.275751    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31d16af-2r9pz" event={"ID":"3994e998-c96e-4c4f-8c5d-6f7f7ce895a8","Type":"ContainerStarted","Data":"5eba27a2d0b257543bea6a18ef077330146272271677554e48292563fd97b8ba"}
Oct 30 21:56:23.275972 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:23.275848    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31d16af-2r9pz" event={"ID":"3994e998-c96e-4c4f-8c5d-6f7f7ce895a8","Type":"ContainerStarted","Data":"738028718a7f99e45cc2c2caf2e1bfcda264205449f6ad9b6771fe5fe6f8eb7f"}
Oct 30 21:56:23.392002 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:23.391954    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31d16af-2r9pz" podStartSLOduration=2.39194101 podStartE2EDuration="2.39194101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:23.388415317 +0000 UTC m=+1377.862919297" watchObservedRunningTime="2026-10-30 21:56:23.39194101 +0000 UTC m=+1377.866444989"
Oct 30 21:56:24.288047 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:24.288006    5087 generic.go:356] "Generic (PLEG): container finished" podID="3994e998-c96e-4c4f-8c5d-6f7f7ce895a8" containerID="5eba27a2d0b257543bea6a18ef077330146272271677554e48292563fd97b8ba" exitCode=0
Oct 30 21:56:24.288240 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:24.288099    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31d16af-2r9pz" event={"ID":"3994e998-c96e-4c4f-8c5d-6f7f7ce895a8","Type":"ContainerDied","Data":"5eba27a2d0b257543bea6a18ef077330146272271677554e48292563fd97b8ba"}
Oct 30 21:56:25.477572 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:25.477544    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31d16af-2r9pz"
Oct 30 21:56:25.598808 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:25.598718    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3994e998-c96e-4c4f-8c5d-6f7f7ce895a8-kube-api-access-w9cvc\" (UniqueName: \"kubernetes.io/projected/3994e998-c96e-4c4f-8c5d-6f7f7ce895a8-kube-api-access-w9cvc\") pod \"3994e998-c96e-4c4f-8c5d-6f7f7ce895a8\" (UID: \"3994e998-c96e-4c4f-8c5d-6f7f7ce895a8\") "
Oct 30 21:56:25.598981 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:25.598807    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3994e998-c96e-4c4f-8c5d-6f7f7ce895a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3994e998-c96e-4c4f-8c5d-6f7f7ce895a8-maas-api-openshift-ca\") pod \"3994e998-c96e-4c4f-8c5d-6f7f7ce895a8\" (UID: \"3994e998-c96e-4c4f-8c5d-6f7f7ce895a8\") "
Oct 30 21:56:25.598981 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:25.598964    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3994e998-c96e-4c4f-8c5d-6f7f7ce895a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3994e998-c96e-4c4f-8c5d-6f7f7ce895a8-maas-api-xks-ca\") pod \"3994e998-c96e-4c4f-8c5d-6f7f7ce895a8\" (UID: \"3994e998-c96e-4c4f-8c5d-6f7f7ce895a8\") "
Oct 30 21:56:25.599677 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:25.599611    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3994e998-c96e-4c4f-8c5d-6f7f7ce895a8-maas-api-openshift-ca" pod "3994e998-c96e-4c4f-8c5d-6f7f7ce895a8" (UID: "3994e998-c96e-4c4f-8c5d-6f7f7ce895a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:25.601773 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:25.601739    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3994e998-c96e-4c4f-8c5d-6f7f7ce895a8-maas-api-xks-ca" pod "3994e998-c96e-4c4f-8c5d-6f7f7ce895a8" (UID: "3994e998-c96e-4c4f-8c5d-6f7f7ce895a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:25.601882 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:25.601741    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3994e998-c96e-4c4f-8c5d-6f7f7ce895a8-kube-api-access-w9cvc" pod "3994e998-c96e-4c4f-8c5d-6f7f7ce895a8" (UID: "3994e998-c96e-4c4f-8c5d-6f7f7ce895a8"). InnerVolumeSpecName "kube-api-access-w9cvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:25.700939 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:25.700912    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9cvc\" (UniqueName: \"kubernetes.io/projected/3994e998-c96e-4c4f-8c5d-6f7f7ce895a8-kube-api-access-w9cvc\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:56:25.700939 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:25.700938    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3994e998-c96e-4c4f-8c5d-6f7f7ce895a8-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:56:25.701127 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:25.700949    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3994e998-c96e-4c4f-8c5d-6f7f7ce895a8-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:56:25.952963 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:25.952880    5087 scope.go:122] "RemoveContainer" containerID="e3c482c32d62c5cbd505672a982ed6569b7ec55004d08f31319c0efbeb313f62"
Oct 30 21:56:25.965446 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:25.965417    5087 scope.go:122] "RemoveContainer" containerID="d72842d63ef93fe019992cc4556956b9f7f8fd2168ccc4339a25d1c87872d3f5"
Oct 30 21:56:25.977894 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:25.977868    5087 scope.go:122] "RemoveContainer" containerID="7ef8a984107848606d82df671d3d0ae0521285d921b2af65b90960ea6cd9bd14"
Oct 30 21:56:25.989881 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:25.989862    5087 scope.go:122] "RemoveContainer" containerID="7de756bb94d594a8d03becc21028f78e5dbc3f5e9d51006ac095a4d09e8a4d2c"
Oct 30 21:56:25.999563 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:25.999542    5087 scope.go:122] "RemoveContainer" containerID="cb3e2d9f0d01640756d6b5d7ffcd1e0ab0fc142b3f33fd45722206de673d986f"
Oct 30 21:56:26.306209 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:26.306174    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31d16af-2r9pz"
Oct 30 21:56:26.306209 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:26.306192    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e31d16af-2r9pz" event={"ID":"3994e998-c96e-4c4f-8c5d-6f7f7ce895a8","Type":"ContainerDied","Data":"738028718a7f99e45cc2c2caf2e1bfcda264205449f6ad9b6771fe5fe6f8eb7f"}
Oct 30 21:56:26.306425 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:26.306264    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="738028718a7f99e45cc2c2caf2e1bfcda264205449f6ad9b6771fe5fe6f8eb7f"
Oct 30 21:56:26.887029 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:26.886996    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d076e07-4x9sn"]
Oct 30 21:56:26.887612 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:26.887598    5087 cpu_manager.go:414] "containerMap: removing container" podUID="3994e998-c96e-4c4f-8c5d-6f7f7ce895a8" containerName="revoke-keys"
Oct 30 21:56:26.887660 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:26.887614    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3994e998-c96e-4c4f-8c5d-6f7f7ce895a8" containerName="revoke-keys"
Oct 30 21:56:26.887728 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:26.887719    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="3994e998-c96e-4c4f-8c5d-6f7f7ce895a8" containerName="revoke-keys"
Oct 30 21:56:26.892961 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:26.892940    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d076e07-4x9sn"
Oct 30 21:56:26.949817 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:26.949789    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d076e07-4x9sn"]
Oct 30 21:56:26.993213 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:26.993188    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:56:26.993403 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:26.993270    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:56:27.019153 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:27.019121    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3c0574e-034f-4e96-80aa-51ca96218b8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d076e07-4x9sn\" (UID: \"f3c0574e-034f-4e96-80aa-51ca96218b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d076e07-4x9sn"
Oct 30 21:56:27.019295 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:27.019240    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48rdp\" (UniqueName: \"kubernetes.io/projected/f3c0574e-034f-4e96-80aa-51ca96218b8c-kube-api-access-48rdp\") pod \"maas-api-revoke-sub-1d076e07-4x9sn\" (UID: \"f3c0574e-034f-4e96-80aa-51ca96218b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d076e07-4x9sn"
Oct 30 21:56:27.019373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:27.019340    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3c0574e-034f-4e96-80aa-51ca96218b8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d076e07-4x9sn\" (UID: \"f3c0574e-034f-4e96-80aa-51ca96218b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d076e07-4x9sn"
Oct 30 21:56:27.120705 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:27.120669    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3c0574e-034f-4e96-80aa-51ca96218b8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d076e07-4x9sn\" (UID: \"f3c0574e-034f-4e96-80aa-51ca96218b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d076e07-4x9sn"
Oct 30 21:56:27.120881 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:27.120743    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48rdp\" (UniqueName: \"kubernetes.io/projected/f3c0574e-034f-4e96-80aa-51ca96218b8c-kube-api-access-48rdp\") pod \"maas-api-revoke-sub-1d076e07-4x9sn\" (UID: \"f3c0574e-034f-4e96-80aa-51ca96218b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d076e07-4x9sn"
Oct 30 21:56:27.120881 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:27.120840    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3c0574e-034f-4e96-80aa-51ca96218b8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d076e07-4x9sn\" (UID: \"f3c0574e-034f-4e96-80aa-51ca96218b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d076e07-4x9sn"
Oct 30 21:56:27.121422 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:27.121340    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3c0574e-034f-4e96-80aa-51ca96218b8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d076e07-4x9sn\" (UID: \"f3c0574e-034f-4e96-80aa-51ca96218b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d076e07-4x9sn"
Oct 30 21:56:27.123629 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:27.123610    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3c0574e-034f-4e96-80aa-51ca96218b8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d076e07-4x9sn\" (UID: \"f3c0574e-034f-4e96-80aa-51ca96218b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d076e07-4x9sn"
Oct 30 21:56:27.178687 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:27.178627    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48rdp\" (UniqueName: \"kubernetes.io/projected/f3c0574e-034f-4e96-80aa-51ca96218b8c-kube-api-access-48rdp\") pod \"maas-api-revoke-sub-1d076e07-4x9sn\" (UID: \"f3c0574e-034f-4e96-80aa-51ca96218b8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d076e07-4x9sn"
Oct 30 21:56:27.206692 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:27.206665    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d076e07-4x9sn"
Oct 30 21:56:27.597739 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:27.597714    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d076e07-4x9sn"]
Oct 30 21:56:27.598795 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:56:27.598770    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3c0574e_034f_4e96_80aa_51ca96218b8c.slice/crio-bc0275f0c35547c3f1b839895cbd92be3abb1e8e8129cb3ee127e867ec7f607a WatchSource:0}: Error finding container bc0275f0c35547c3f1b839895cbd92be3abb1e8e8129cb3ee127e867ec7f607a: Status 404 returned error can't find the container with id bc0275f0c35547c3f1b839895cbd92be3abb1e8e8129cb3ee127e867ec7f607a
Oct 30 21:56:28.330643 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:28.330591    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d076e07-4x9sn" event={"ID":"f3c0574e-034f-4e96-80aa-51ca96218b8c","Type":"ContainerStarted","Data":"bdda87aa1ec77092e81dc0a78d2aed1a90a73cc02f6978486f739b9fe1169495"}
Oct 30 21:56:28.331107 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:28.330695    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d076e07-4x9sn" event={"ID":"f3c0574e-034f-4e96-80aa-51ca96218b8c","Type":"ContainerStarted","Data":"bc0275f0c35547c3f1b839895cbd92be3abb1e8e8129cb3ee127e867ec7f607a"}
Oct 30 21:56:28.450572 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:28.450526    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d076e07-4x9sn" podStartSLOduration=2.450512935 podStartE2EDuration="2.450512935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:28.445555233 +0000 UTC m=+1382.920059213" watchObservedRunningTime="2026-10-30 21:56:28.450512935 +0000 UTC m=+1382.925016945"
Oct 30 21:56:29.339652 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:29.339616    5087 generic.go:356] "Generic (PLEG): container finished" podID="f3c0574e-034f-4e96-80aa-51ca96218b8c" containerID="bdda87aa1ec77092e81dc0a78d2aed1a90a73cc02f6978486f739b9fe1169495" exitCode=0
Oct 30 21:56:29.340055 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:29.339706    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d076e07-4x9sn" event={"ID":"f3c0574e-034f-4e96-80aa-51ca96218b8c","Type":"ContainerDied","Data":"bdda87aa1ec77092e81dc0a78d2aed1a90a73cc02f6978486f739b9fe1169495"}
Oct 30 21:56:30.530285 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:30.530262    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d076e07-4x9sn"
Oct 30 21:56:30.672696 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:30.672608    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3c0574e-034f-4e96-80aa-51ca96218b8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3c0574e-034f-4e96-80aa-51ca96218b8c-maas-api-openshift-ca\") pod \"f3c0574e-034f-4e96-80aa-51ca96218b8c\" (UID: \"f3c0574e-034f-4e96-80aa-51ca96218b8c\") "
Oct 30 21:56:30.672862 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:30.672755    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3c0574e-034f-4e96-80aa-51ca96218b8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3c0574e-034f-4e96-80aa-51ca96218b8c-maas-api-xks-ca\") pod \"f3c0574e-034f-4e96-80aa-51ca96218b8c\" (UID: \"f3c0574e-034f-4e96-80aa-51ca96218b8c\") "
Oct 30 21:56:30.672862 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:30.672780    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3c0574e-034f-4e96-80aa-51ca96218b8c-kube-api-access-48rdp\" (UniqueName: \"kubernetes.io/projected/f3c0574e-034f-4e96-80aa-51ca96218b8c-kube-api-access-48rdp\") pod \"f3c0574e-034f-4e96-80aa-51ca96218b8c\" (UID: \"f3c0574e-034f-4e96-80aa-51ca96218b8c\") "
Oct 30 21:56:30.673415 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:30.673386    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3c0574e-034f-4e96-80aa-51ca96218b8c-maas-api-openshift-ca" pod "f3c0574e-034f-4e96-80aa-51ca96218b8c" (UID: "f3c0574e-034f-4e96-80aa-51ca96218b8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:30.675554 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:30.675526    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3c0574e-034f-4e96-80aa-51ca96218b8c-maas-api-xks-ca" pod "f3c0574e-034f-4e96-80aa-51ca96218b8c" (UID: "f3c0574e-034f-4e96-80aa-51ca96218b8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:30.675793 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:30.675770    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3c0574e-034f-4e96-80aa-51ca96218b8c-kube-api-access-48rdp" pod "f3c0574e-034f-4e96-80aa-51ca96218b8c" (UID: "f3c0574e-034f-4e96-80aa-51ca96218b8c"). InnerVolumeSpecName "kube-api-access-48rdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:30.774987 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:30.774948    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3c0574e-034f-4e96-80aa-51ca96218b8c-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:56:30.774987 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:30.774986    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48rdp\" (UniqueName: \"kubernetes.io/projected/f3c0574e-034f-4e96-80aa-51ca96218b8c-kube-api-access-48rdp\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:56:30.775193 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:30.775004    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3c0574e-034f-4e96-80aa-51ca96218b8c-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:56:31.259761 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:31.259729    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5521b849-8gr5m"]
Oct 30 21:56:31.260417 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:31.260397    5087 cpu_manager.go:414] "containerMap: removing container" podUID="f3c0574e-034f-4e96-80aa-51ca96218b8c" containerName="revoke-keys"
Oct 30 21:56:31.260417 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:31.260418    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3c0574e-034f-4e96-80aa-51ca96218b8c" containerName="revoke-keys"
Oct 30 21:56:31.260637 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:31.260623    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3c0574e-034f-4e96-80aa-51ca96218b8c" containerName="revoke-keys"
Oct 30 21:56:31.264425 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:31.264405    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5521b849-8gr5m"
Oct 30 21:56:31.372110 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:31.372076    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d076e07-4x9sn" event={"ID":"f3c0574e-034f-4e96-80aa-51ca96218b8c","Type":"ContainerDied","Data":"bc0275f0c35547c3f1b839895cbd92be3abb1e8e8129cb3ee127e867ec7f607a"}
Oct 30 21:56:31.372110 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:31.372099    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d076e07-4x9sn"
Oct 30 21:56:31.372326 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:31.372131    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc0275f0c35547c3f1b839895cbd92be3abb1e8e8129cb3ee127e867ec7f607a"
Oct 30 21:56:31.375304 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:31.374988    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5521b849-8gr5m"]
Oct 30 21:56:31.387483 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:31.386609    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5521b849-8gr5m\" (UID: \"d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5521b849-8gr5m"
Oct 30 21:56:31.387483 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:31.386815    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9cd7\" (UniqueName: \"kubernetes.io/projected/d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630-kube-api-access-v9cd7\") pod \"maas-api-revoke-sub-5521b849-8gr5m\" (UID: \"d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5521b849-8gr5m"
Oct 30 21:56:31.387483 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:31.386975    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5521b849-8gr5m\" (UID: \"d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5521b849-8gr5m"
Oct 30 21:56:31.488813 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:31.488775    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9cd7\" (UniqueName: \"kubernetes.io/projected/d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630-kube-api-access-v9cd7\") pod \"maas-api-revoke-sub-5521b849-8gr5m\" (UID: \"d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5521b849-8gr5m"
Oct 30 21:56:31.489005 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:31.488874    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5521b849-8gr5m\" (UID: \"d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5521b849-8gr5m"
Oct 30 21:56:31.489005 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:31.488942    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5521b849-8gr5m\" (UID: \"d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5521b849-8gr5m"
Oct 30 21:56:31.489653 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:31.489630    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5521b849-8gr5m\" (UID: \"d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5521b849-8gr5m"
Oct 30 21:56:31.491486 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:31.491464    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5521b849-8gr5m\" (UID: \"d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5521b849-8gr5m"
Oct 30 21:56:31.546868 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:31.546808    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9cd7\" (UniqueName: \"kubernetes.io/projected/d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630-kube-api-access-v9cd7\") pod \"maas-api-revoke-sub-5521b849-8gr5m\" (UID: \"d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5521b849-8gr5m"
Oct 30 21:56:31.579732 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:31.579710    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5521b849-8gr5m"
Oct 30 21:56:31.959806 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:31.959777    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5521b849-8gr5m"]
Oct 30 21:56:31.960538 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:56:31.960515    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ef26f0_5d2e_4ca5_8d34_bbd32bf58630.slice/crio-2dcc7945c3f4906e2a3380d00fe527800d6a3495c15f721edaec2bcc772d30dc WatchSource:0}: Error finding container 2dcc7945c3f4906e2a3380d00fe527800d6a3495c15f721edaec2bcc772d30dc: Status 404 returned error can't find the container with id 2dcc7945c3f4906e2a3380d00fe527800d6a3495c15f721edaec2bcc772d30dc
Oct 30 21:56:32.380888 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:32.380858    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5521b849-8gr5m" event={"ID":"d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630","Type":"ContainerStarted","Data":"ecc77fe7500e2e06de3011141f0c3a08f834daf0dcb15d595a2babe5644a7182"}
Oct 30 21:56:32.381056 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:32.380907    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5521b849-8gr5m" event={"ID":"d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630","Type":"ContainerStarted","Data":"2dcc7945c3f4906e2a3380d00fe527800d6a3495c15f721edaec2bcc772d30dc"}
Oct 30 21:56:32.502794 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:32.502693    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5521b849-8gr5m" podStartSLOduration=1.502679508 podStartE2EDuration="1.502679508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:32.500121864 +0000 UTC m=+1386.974625844" watchObservedRunningTime="2026-10-30 21:56:32.502679508 +0000 UTC m=+1386.977183465"
Oct 30 21:56:32.645934 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:32.645904    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffde7781-5rs9s"]
Oct 30 21:56:32.650305 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:32.650285    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffde7781-5rs9s"
Oct 30 21:56:32.707014 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:32.706991    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffde7781-5rs9s"]
Oct 30 21:56:32.807372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:32.807276    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g24v\" (UniqueName: \"kubernetes.io/projected/2decf802-36b5-4585-b32d-6a917c82e0ac-kube-api-access-6g24v\") pod \"maas-api-revoke-sub-ffde7781-5rs9s\" (UID: \"2decf802-36b5-4585-b32d-6a917c82e0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffde7781-5rs9s"
Oct 30 21:56:32.807372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:32.807317    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2decf802-36b5-4585-b32d-6a917c82e0ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffde7781-5rs9s\" (UID: \"2decf802-36b5-4585-b32d-6a917c82e0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffde7781-5rs9s"
Oct 30 21:56:32.807610 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:32.807587    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2decf802-36b5-4585-b32d-6a917c82e0ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffde7781-5rs9s\" (UID: \"2decf802-36b5-4585-b32d-6a917c82e0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffde7781-5rs9s"
Oct 30 21:56:32.908964 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:32.908938    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2decf802-36b5-4585-b32d-6a917c82e0ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffde7781-5rs9s\" (UID: \"2decf802-36b5-4585-b32d-6a917c82e0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffde7781-5rs9s"
Oct 30 21:56:32.909181 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:32.909161    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g24v\" (UniqueName: \"kubernetes.io/projected/2decf802-36b5-4585-b32d-6a917c82e0ac-kube-api-access-6g24v\") pod \"maas-api-revoke-sub-ffde7781-5rs9s\" (UID: \"2decf802-36b5-4585-b32d-6a917c82e0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffde7781-5rs9s"
Oct 30 21:56:32.909278 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:32.909208    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2decf802-36b5-4585-b32d-6a917c82e0ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffde7781-5rs9s\" (UID: \"2decf802-36b5-4585-b32d-6a917c82e0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffde7781-5rs9s"
Oct 30 21:56:32.909897 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:32.909875    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2decf802-36b5-4585-b32d-6a917c82e0ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ffde7781-5rs9s\" (UID: \"2decf802-36b5-4585-b32d-6a917c82e0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffde7781-5rs9s"
Oct 30 21:56:32.911718 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:32.911689    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2decf802-36b5-4585-b32d-6a917c82e0ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ffde7781-5rs9s\" (UID: \"2decf802-36b5-4585-b32d-6a917c82e0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffde7781-5rs9s"
Oct 30 21:56:32.965890 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:32.965867    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g24v\" (UniqueName: \"kubernetes.io/projected/2decf802-36b5-4585-b32d-6a917c82e0ac-kube-api-access-6g24v\") pod \"maas-api-revoke-sub-ffde7781-5rs9s\" (UID: \"2decf802-36b5-4585-b32d-6a917c82e0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffde7781-5rs9s"
Oct 30 21:56:33.263769 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:33.263731    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffde7781-5rs9s"
Oct 30 21:56:33.395408 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:33.392384    5087 generic.go:356] "Generic (PLEG): container finished" podID="d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630" containerID="ecc77fe7500e2e06de3011141f0c3a08f834daf0dcb15d595a2babe5644a7182" exitCode=0
Oct 30 21:56:33.395408 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:33.392443    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5521b849-8gr5m" event={"ID":"d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630","Type":"ContainerDied","Data":"ecc77fe7500e2e06de3011141f0c3a08f834daf0dcb15d595a2babe5644a7182"}
Oct 30 21:56:33.648195 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:33.648167    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffde7781-5rs9s"]
Oct 30 21:56:33.649432 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:56:33.649404    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2decf802_36b5_4585_b32d_6a917c82e0ac.slice/crio-7a8dea07315da9ba9709c25aca5fd67d0c607d0fb37f5836d349e9215534c0b3 WatchSource:0}: Error finding container 7a8dea07315da9ba9709c25aca5fd67d0c607d0fb37f5836d349e9215534c0b3: Status 404 returned error can't find the container with id 7a8dea07315da9ba9709c25aca5fd67d0c607d0fb37f5836d349e9215534c0b3
Oct 30 21:56:34.051307 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:34.051274    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5aa62eb-rw7kj"]
Oct 30 21:56:34.055321 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:34.055293    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5aa62eb-rw7kj"]
Oct 30 21:56:34.402557 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:34.402467    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffde7781-5rs9s" event={"ID":"2decf802-36b5-4585-b32d-6a917c82e0ac","Type":"ContainerStarted","Data":"d2836016f17582522aa1dcc5fdd9edc25c6cc763821ab994c7e64e69842b47d3"}
Oct 30 21:56:34.402557 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:34.402529    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffde7781-5rs9s" event={"ID":"2decf802-36b5-4585-b32d-6a917c82e0ac","Type":"ContainerStarted","Data":"7a8dea07315da9ba9709c25aca5fd67d0c607d0fb37f5836d349e9215534c0b3"}
Oct 30 21:56:34.521185 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:34.521121    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffde7781-5rs9s" podStartSLOduration=2.521101347 podStartE2EDuration="2.521101347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:34.51975587 +0000 UTC m=+1388.994259849" watchObservedRunningTime="2026-10-30 21:56:34.521101347 +0000 UTC m=+1388.995605328"
Oct 30 21:56:34.632761 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:34.632729    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5521b849-8gr5m"
Oct 30 21:56:34.636228 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:34.636201    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630-kube-api-access-v9cd7\" (UniqueName: \"kubernetes.io/projected/d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630-kube-api-access-v9cd7\") pod \"d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630\" (UID: \"d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630\") "
Oct 30 21:56:34.636389 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:34.636281    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630-maas-api-xks-ca\") pod \"d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630\" (UID: \"d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630\") "
Oct 30 21:56:34.636389 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:34.636325    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630-maas-api-openshift-ca\") pod \"d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630\" (UID: \"d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630\") "
Oct 30 21:56:34.636688 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:34.636663    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630-maas-api-openshift-ca" pod "d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630" (UID: "d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:34.636859 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:34.636840    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:56:34.638255 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:34.638228    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630-kube-api-access-v9cd7" pod "d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630" (UID: "d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630"). InnerVolumeSpecName "kube-api-access-v9cd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:34.638382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:34.638310    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630-maas-api-xks-ca" pod "d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630" (UID: "d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:34.738127 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:34.738094    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9cd7\" (UniqueName: \"kubernetes.io/projected/d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630-kube-api-access-v9cd7\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:56:34.738529 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:34.738130    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:56:35.411204 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:35.411174    5087 generic.go:356] "Generic (PLEG): container finished" podID="2decf802-36b5-4585-b32d-6a917c82e0ac" containerID="d2836016f17582522aa1dcc5fdd9edc25c6cc763821ab994c7e64e69842b47d3" exitCode=0
Oct 30 21:56:35.411440 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:35.411263    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffde7781-5rs9s" event={"ID":"2decf802-36b5-4585-b32d-6a917c82e0ac","Type":"ContainerDied","Data":"d2836016f17582522aa1dcc5fdd9edc25c6cc763821ab994c7e64e69842b47d3"}
Oct 30 21:56:35.412689 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:35.412665    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5521b849-8gr5m" event={"ID":"d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630","Type":"ContainerDied","Data":"2dcc7945c3f4906e2a3380d00fe527800d6a3495c15f721edaec2bcc772d30dc"}
Oct 30 21:56:35.412796 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:35.412709    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dcc7945c3f4906e2a3380d00fe527800d6a3495c15f721edaec2bcc772d30dc"
Oct 30 21:56:35.412796 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:35.412725    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5521b849-8gr5m"
Oct 30 21:56:35.710732 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:35.710657    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d82ef9ca-3cfa-4141-8669-af559a5384c0" path="/var/lib/kubelet/pods/d82ef9ca-3cfa-4141-8669-af559a5384c0/volumes"
Oct 30 21:56:36.622751 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:36.622720    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffde7781-5rs9s"
Oct 30 21:56:36.662631 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:36.662595    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2decf802-36b5-4585-b32d-6a917c82e0ac-kube-api-access-6g24v\" (UniqueName: \"kubernetes.io/projected/2decf802-36b5-4585-b32d-6a917c82e0ac-kube-api-access-6g24v\") pod \"2decf802-36b5-4585-b32d-6a917c82e0ac\" (UID: \"2decf802-36b5-4585-b32d-6a917c82e0ac\") "
Oct 30 21:56:36.662818 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:36.662773    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2decf802-36b5-4585-b32d-6a917c82e0ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2decf802-36b5-4585-b32d-6a917c82e0ac-maas-api-openshift-ca\") pod \"2decf802-36b5-4585-b32d-6a917c82e0ac\" (UID: \"2decf802-36b5-4585-b32d-6a917c82e0ac\") "
Oct 30 21:56:36.662958 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:36.662921    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2decf802-36b5-4585-b32d-6a917c82e0ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2decf802-36b5-4585-b32d-6a917c82e0ac-maas-api-xks-ca\") pod \"2decf802-36b5-4585-b32d-6a917c82e0ac\" (UID: \"2decf802-36b5-4585-b32d-6a917c82e0ac\") "
Oct 30 21:56:36.663157 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:36.663129    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2decf802-36b5-4585-b32d-6a917c82e0ac-maas-api-openshift-ca" pod "2decf802-36b5-4585-b32d-6a917c82e0ac" (UID: "2decf802-36b5-4585-b32d-6a917c82e0ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:36.663579 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:36.663553    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2decf802-36b5-4585-b32d-6a917c82e0ac-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:56:36.665185 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:36.665156    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2decf802-36b5-4585-b32d-6a917c82e0ac-kube-api-access-6g24v" pod "2decf802-36b5-4585-b32d-6a917c82e0ac" (UID: "2decf802-36b5-4585-b32d-6a917c82e0ac"). InnerVolumeSpecName "kube-api-access-6g24v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:36.665262 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:36.665175    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2decf802-36b5-4585-b32d-6a917c82e0ac-maas-api-xks-ca" pod "2decf802-36b5-4585-b32d-6a917c82e0ac" (UID: "2decf802-36b5-4585-b32d-6a917c82e0ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:36.764967 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:36.764931    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2decf802-36b5-4585-b32d-6a917c82e0ac-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:56:36.764967 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:36.764968    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g24v\" (UniqueName: \"kubernetes.io/projected/2decf802-36b5-4585-b32d-6a917c82e0ac-kube-api-access-6g24v\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:56:37.439414 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:37.439386    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffde7781-5rs9s"
Oct 30 21:56:37.439580 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:37.439383    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ffde7781-5rs9s" event={"ID":"2decf802-36b5-4585-b32d-6a917c82e0ac","Type":"ContainerDied","Data":"7a8dea07315da9ba9709c25aca5fd67d0c607d0fb37f5836d349e9215534c0b3"}
Oct 30 21:56:37.439580 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:37.439508    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a8dea07315da9ba9709c25aca5fd67d0c607d0fb37f5836d349e9215534c0b3"
Oct 30 21:56:42.366066 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:42.366024    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b4645f5-mmmfg"]
Oct 30 21:56:42.366827 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:42.366804    5087 cpu_manager.go:414] "containerMap: removing container" podUID="2decf802-36b5-4585-b32d-6a917c82e0ac" containerName="revoke-keys"
Oct 30 21:56:42.366926 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:42.366828    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2decf802-36b5-4585-b32d-6a917c82e0ac" containerName="revoke-keys"
Oct 30 21:56:42.366926 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:42.366910    5087 cpu_manager.go:414] "containerMap: removing container" podUID="d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630" containerName="revoke-keys"
Oct 30 21:56:42.366926 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:42.366919    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630" containerName="revoke-keys"
Oct 30 21:56:42.367091 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:42.367076    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630" containerName="revoke-keys"
Oct 30 21:56:42.367142 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:42.367101    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="2decf802-36b5-4585-b32d-6a917c82e0ac" containerName="revoke-keys"
Oct 30 21:56:42.370702 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:42.370678    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4645f5-mmmfg"
Oct 30 21:56:42.434979 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:42.434948    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b4645f5-mmmfg"]
Oct 30 21:56:42.443606 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:42.443569    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2mqs\" (UniqueName: \"kubernetes.io/projected/40d91704-2d27-4030-b79e-74046213e9d6-kube-api-access-h2mqs\") pod \"maas-api-revoke-sub-2b4645f5-mmmfg\" (UID: \"40d91704-2d27-4030-b79e-74046213e9d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4645f5-mmmfg"
Oct 30 21:56:42.443862 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:42.443636    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40d91704-2d27-4030-b79e-74046213e9d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b4645f5-mmmfg\" (UID: \"40d91704-2d27-4030-b79e-74046213e9d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4645f5-mmmfg"
Oct 30 21:56:42.443862 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:42.443660    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40d91704-2d27-4030-b79e-74046213e9d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b4645f5-mmmfg\" (UID: \"40d91704-2d27-4030-b79e-74046213e9d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4645f5-mmmfg"
Oct 30 21:56:42.479193 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:42.479162    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:56:42.479422 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:42.479405    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:56:42.545124 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:42.545095    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2mqs\" (UniqueName: \"kubernetes.io/projected/40d91704-2d27-4030-b79e-74046213e9d6-kube-api-access-h2mqs\") pod \"maas-api-revoke-sub-2b4645f5-mmmfg\" (UID: \"40d91704-2d27-4030-b79e-74046213e9d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4645f5-mmmfg"
Oct 30 21:56:42.545436 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:42.545139    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40d91704-2d27-4030-b79e-74046213e9d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b4645f5-mmmfg\" (UID: \"40d91704-2d27-4030-b79e-74046213e9d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4645f5-mmmfg"
Oct 30 21:56:42.545436 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:42.545162    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40d91704-2d27-4030-b79e-74046213e9d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b4645f5-mmmfg\" (UID: \"40d91704-2d27-4030-b79e-74046213e9d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4645f5-mmmfg"
Oct 30 21:56:42.546286 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:42.546261    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40d91704-2d27-4030-b79e-74046213e9d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b4645f5-mmmfg\" (UID: \"40d91704-2d27-4030-b79e-74046213e9d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4645f5-mmmfg"
Oct 30 21:56:42.547952 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:42.547926    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40d91704-2d27-4030-b79e-74046213e9d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b4645f5-mmmfg\" (UID: \"40d91704-2d27-4030-b79e-74046213e9d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4645f5-mmmfg"
Oct 30 21:56:42.601623 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:42.601595    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2mqs\" (UniqueName: \"kubernetes.io/projected/40d91704-2d27-4030-b79e-74046213e9d6-kube-api-access-h2mqs\") pod \"maas-api-revoke-sub-2b4645f5-mmmfg\" (UID: \"40d91704-2d27-4030-b79e-74046213e9d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4645f5-mmmfg"
Oct 30 21:56:42.685617 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:42.685537    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4645f5-mmmfg"
Oct 30 21:56:43.065863 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:43.065833    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b4645f5-mmmfg"]
Oct 30 21:56:43.066227 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:56:43.066204    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d91704_2d27_4030_b79e_74046213e9d6.slice/crio-ad56a5184f817ba897593de7a4821990f627dbfee95dc739ee8d7b23ecb8d2e9 WatchSource:0}: Error finding container ad56a5184f817ba897593de7a4821990f627dbfee95dc739ee8d7b23ecb8d2e9: Status 404 returned error can't find the container with id ad56a5184f817ba897593de7a4821990f627dbfee95dc739ee8d7b23ecb8d2e9
Oct 30 21:56:43.502600 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:43.502561    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4645f5-mmmfg" event={"ID":"40d91704-2d27-4030-b79e-74046213e9d6","Type":"ContainerStarted","Data":"3639c6b19a3071c56e64202b85e9cba57da9c623824606c9458e5db91c281a59"}
Oct 30 21:56:43.503062 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:43.502614    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4645f5-mmmfg" event={"ID":"40d91704-2d27-4030-b79e-74046213e9d6","Type":"ContainerStarted","Data":"ad56a5184f817ba897593de7a4821990f627dbfee95dc739ee8d7b23ecb8d2e9"}
Oct 30 21:56:43.625450 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:43.625172    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4645f5-mmmfg" podStartSLOduration=1.6251600449999999 podStartE2EDuration="1.625160045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:43.625076101 +0000 UTC m=+1398.099580080" watchObservedRunningTime="2026-10-30 21:56:43.625160045 +0000 UTC m=+1398.099664024"
Oct 30 21:56:44.512907 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:44.512865    5087 generic.go:356] "Generic (PLEG): container finished" podID="40d91704-2d27-4030-b79e-74046213e9d6" containerID="3639c6b19a3071c56e64202b85e9cba57da9c623824606c9458e5db91c281a59" exitCode=0
Oct 30 21:56:44.513252 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:44.512970    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4645f5-mmmfg" event={"ID":"40d91704-2d27-4030-b79e-74046213e9d6","Type":"ContainerDied","Data":"3639c6b19a3071c56e64202b85e9cba57da9c623824606c9458e5db91c281a59"}
Oct 30 21:56:44.896959 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:44.896879    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"]
Oct 30 21:56:44.901209 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:44.901194    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:56:44.978918 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:44.978878    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istio-token\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:56:44.979098 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:44.978935    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-workload-socket\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:56:44.979098 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:44.979076    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istio-envoy\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:56:44.979224 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:44.979174    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-workload-certs\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:56:44.979224 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:44.979202    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istio-data\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:56:44.979339 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:44.979254    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-credential-socket\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:56:44.979339 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:44.979324    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istio-podinfo\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:56:44.979463 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:44.979435    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istiod-ca-cert\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:56:44.979587 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:44.979571    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q86d\" (UniqueName: \"kubernetes.io/projected/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-kube-api-access-7q86d\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:56:45.009510 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.009481    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"]
Oct 30 21:56:45.081190 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.081161    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istio-token\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:56:45.081376 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.081200    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-workload-socket\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:56:45.081376 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.081229    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istio-envoy\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:56:45.081376 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.081306    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-workload-certs\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:56:45.081376 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.081329    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istio-data\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:56:45.081619 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.081394    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-credential-socket\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:56:45.081619 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.081450    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istio-podinfo\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:56:45.081619 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.081527    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istiod-ca-cert\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:56:45.081784 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.081665    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-workload-socket\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:56:45.081784 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.081690    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q86d\" (UniqueName: \"kubernetes.io/projected/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-kube-api-access-7q86d\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:56:45.081784 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.081730    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-workload-certs\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:56:45.081784 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.081775    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-credential-socket\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:56:45.082009 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.081919    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istio-data\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:56:45.082417 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.082388    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istiod-ca-cert\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:56:45.083967 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.083940    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istio-envoy\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:56:45.084337 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.084313    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istio-podinfo\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:56:45.139233 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.139174    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istio-token\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:56:45.140021 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.139994    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q86d\" (UniqueName: \"kubernetes.io/projected/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-kube-api-access-7q86d\") pod \"e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") " pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:56:45.175976 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.175912    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"]
Oct 30 21:56:45.181708 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.181688    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:56:45.215077 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.215053    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:56:45.229618 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.229598    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"]
Oct 30 21:56:45.233920 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.233905    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:56:45.253919 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.253885    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"]
Oct 30 21:56:45.291168 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.286159    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istio-podinfo\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:56:45.291168 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.286270    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istio-data\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:56:45.291168 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.286383    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istiod-ca-cert\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:56:45.291168 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.286445    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-workload-socket\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:56:45.291168 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.286479    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ctr4\" (UniqueName: \"kubernetes.io/projected/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-kube-api-access-6ctr4\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:56:45.291168 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.286516    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-credential-socket\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:56:45.292422 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.292301    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istio-token\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:56:45.292710 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.292681    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-workload-certs\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:56:45.292814 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.292764    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istio-envoy\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:56:45.321913 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.321831    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"]
Oct 30 21:56:45.394495 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.394467    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istio-podinfo\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:56:45.394675 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.394523    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b7cb87-8d56-4822-9d11-1cd8185c2732-istiod-ca-cert\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:56:45.394675 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.394581    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istio-data\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:56:45.394675 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.394632    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-credential-socket\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:56:45.394856 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.394723    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b7cb87-8d56-4822-9d11-1cd8185c2732-istio-podinfo\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:56:45.394856 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.394774    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istiod-ca-cert\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:56:45.394856 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.394834    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-workload-socket\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:56:45.395024 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.394868    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ctr4\" (UniqueName: \"kubernetes.io/projected/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-kube-api-access-6ctr4\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:56:45.395024 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.394906    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-credential-socket\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:56:45.395024 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.394934    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istio-token\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:56:45.395024 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.394983    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-istio-envoy\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:56:45.395024 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.395008    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-workload-socket\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:56:45.395280 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.395029    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhcjt\" (UniqueName: \"kubernetes.io/projected/14b7cb87-8d56-4822-9d11-1cd8185c2732-kube-api-access-lhcjt\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:56:45.395280 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.395064    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-workload-certs\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:56:45.395280 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.395080    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istio-data\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:56:45.395280 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.395088    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-workload-certs\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:56:45.395280 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.395145    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istio-envoy\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:56:45.395280 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.395164    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-istio-data\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:56:45.395280 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.395221    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14b7cb87-8d56-4822-9d11-1cd8185c2732-istio-token\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:56:45.395999 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.395975    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-workload-socket\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:56:45.395999 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.395990    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istiod-ca-cert\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:56:45.396266 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.396244    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-workload-certs\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:56:45.396521 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.396498    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-credential-socket\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:56:45.398828 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.398784    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istio-envoy\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:56:45.398931 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.398890    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istio-podinfo\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:56:45.452470 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.452364    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istio-token\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:56:45.452587 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.452498    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ctr4\" (UniqueName: \"kubernetes.io/projected/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-kube-api-access-6ctr4\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:56:45.497304 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.497275    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-workload-certs\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:56:45.497490 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.497321    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-istio-data\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:56:45.497490 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.497410    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14b7cb87-8d56-4822-9d11-1cd8185c2732-istio-token\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:56:45.497490 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.497473    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b7cb87-8d56-4822-9d11-1cd8185c2732-istiod-ca-cert\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:56:45.497669 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.497557    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-credential-socket\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:56:45.497949 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.497594    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:56:45.498033 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.497599    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b7cb87-8d56-4822-9d11-1cd8185c2732-istio-podinfo\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:56:45.498088 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.498056    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b7cb87-8d56-4822-9d11-1cd8185c2732-istiod-ca-cert\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:56:45.498088 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.497666    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-workload-certs\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:56:45.498088 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.497717    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-istio-data\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:56:45.498209 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.497910    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-credential-socket\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:56:45.498209 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.498183    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-istio-envoy\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:56:45.498209 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.498206    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-workload-socket\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:56:45.498331 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.498223    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhcjt\" (UniqueName: \"kubernetes.io/projected/14b7cb87-8d56-4822-9d11-1cd8185c2732-kube-api-access-lhcjt\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:56:45.498615 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.498582    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-workload-socket\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:56:45.500632 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.500610    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-istio-envoy\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:56:45.500752 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.500737    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b7cb87-8d56-4822-9d11-1cd8185c2732-istio-podinfo\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:56:45.554479 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.554422    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14b7cb87-8d56-4822-9d11-1cd8185c2732-istio-token\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:56:45.556116 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.556080    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhcjt\" (UniqueName: \"kubernetes.io/projected/14b7cb87-8d56-4822-9d11-1cd8185c2732-kube-api-access-lhcjt\") pod \"e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") " pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:56:45.587175 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.583258    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:56:45.666691 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.666504    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"]
Oct 30 21:56:45.680396 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.679527    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:56:45.680396 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.679615    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:56:45.680396 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.679660    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:56:45.781841 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.781815    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4645f5-mmmfg"
Oct 30 21:56:45.905808 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.905780    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"]
Oct 30 21:56:45.906738 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:56:45.906709    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09a8a5cd_2a5e_4afb_9aae_7680fedb0c2b.slice/crio-0d757e359f657936ce471fe84368492e9dae16c3427a6e32fea742c7e2e1d2bf WatchSource:0}: Error finding container 0d757e359f657936ce471fe84368492e9dae16c3427a6e32fea742c7e2e1d2bf: Status 404 returned error can't find the container with id 0d757e359f657936ce471fe84368492e9dae16c3427a6e32fea742c7e2e1d2bf
Oct 30 21:56:45.909735 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.909700    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:56:45.909851 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.909792    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:56:45.909851 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.909834    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:56:45.912639 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.912618    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40d91704-2d27-4030-b79e-74046213e9d6-kube-api-access-h2mqs\" (UniqueName: \"kubernetes.io/projected/40d91704-2d27-4030-b79e-74046213e9d6-kube-api-access-h2mqs\") pod \"40d91704-2d27-4030-b79e-74046213e9d6\" (UID: \"40d91704-2d27-4030-b79e-74046213e9d6\") "
Oct 30 21:56:45.914608 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.912845    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40d91704-2d27-4030-b79e-74046213e9d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40d91704-2d27-4030-b79e-74046213e9d6-maas-api-xks-ca\") pod \"40d91704-2d27-4030-b79e-74046213e9d6\" (UID: \"40d91704-2d27-4030-b79e-74046213e9d6\") "
Oct 30 21:56:45.914608 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.913194    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40d91704-2d27-4030-b79e-74046213e9d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40d91704-2d27-4030-b79e-74046213e9d6-maas-api-openshift-ca\") pod \"40d91704-2d27-4030-b79e-74046213e9d6\" (UID: \"40d91704-2d27-4030-b79e-74046213e9d6\") "
Oct 30 21:56:45.914608 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.913793    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d91704-2d27-4030-b79e-74046213e9d6-maas-api-openshift-ca" pod "40d91704-2d27-4030-b79e-74046213e9d6" (UID: "40d91704-2d27-4030-b79e-74046213e9d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:56:45.914608 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.914302    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40d91704-2d27-4030-b79e-74046213e9d6-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:56:45.915572 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.915428    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40d91704-2d27-4030-b79e-74046213e9d6-maas-api-xks-ca" pod "40d91704-2d27-4030-b79e-74046213e9d6" (UID: "40d91704-2d27-4030-b79e-74046213e9d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:56:45.916042 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:45.916020    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d91704-2d27-4030-b79e-74046213e9d6-kube-api-access-h2mqs" pod "40d91704-2d27-4030-b79e-74046213e9d6" (UID: "40d91704-2d27-4030-b79e-74046213e9d6"). InnerVolumeSpecName "kube-api-access-h2mqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:56:46.003781 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:46.003702    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"]
Oct 30 21:56:46.004220 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:56:46.004198    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b7cb87_8d56_4822_9d11_1cd8185c2732.slice/crio-1335464d56cdf8d67d77930d1ad40f0e0413036decd4e48b5ec072a500d2e237 WatchSource:0}: Error finding container 1335464d56cdf8d67d77930d1ad40f0e0413036decd4e48b5ec072a500d2e237: Status 404 returned error can't find the container with id 1335464d56cdf8d67d77930d1ad40f0e0413036decd4e48b5ec072a500d2e237
Oct 30 21:56:46.006872 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:46.006839    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:56:46.006949 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:46.006933    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:56:46.006990 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:46.006979    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:56:46.015500 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:46.015472    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2mqs\" (UniqueName: \"kubernetes.io/projected/40d91704-2d27-4030-b79e-74046213e9d6-kube-api-access-h2mqs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:56:46.015594 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:46.015505    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40d91704-2d27-4030-b79e-74046213e9d6-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:56:46.542366 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:46.542026    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b" event={"ID":"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b","Type":"ContainerStarted","Data":"b31aa2bcb35339be1dff9831eebc09071b1dc7c70b0e9f49eaef7fb2e4bbe60b"}
Oct 30 21:56:46.542366 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:46.542132    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b" event={"ID":"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b","Type":"ContainerStarted","Data":"0d757e359f657936ce471fe84368492e9dae16c3427a6e32fea742c7e2e1d2bf"}
Oct 30 21:56:46.546536 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:46.546509    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c" event={"ID":"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d","Type":"ContainerStarted","Data":"6096c0cb37cd14bf780ed09175ea098d94b23127b945eb763680406d75afcd51"}
Oct 30 21:56:46.547227 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:46.547206    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c" event={"ID":"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d","Type":"ContainerStarted","Data":"553764ee19e2424d7c061141db243e740aef22761e77295f045df1a219e68c08"}
Oct 30 21:56:46.550408 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:46.550387    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb" event={"ID":"14b7cb87-8d56-4822-9d11-1cd8185c2732","Type":"ContainerStarted","Data":"cc78eda6b924355be42e10243a8354e8a89f0f2673094a96e1c6941913041f7f"}
Oct 30 21:56:46.550551 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:46.550537    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb" event={"ID":"14b7cb87-8d56-4822-9d11-1cd8185c2732","Type":"ContainerStarted","Data":"1335464d56cdf8d67d77930d1ad40f0e0413036decd4e48b5ec072a500d2e237"}
Oct 30 21:56:46.552814 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:46.552794    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4645f5-mmmfg" event={"ID":"40d91704-2d27-4030-b79e-74046213e9d6","Type":"ContainerDied","Data":"ad56a5184f817ba897593de7a4821990f627dbfee95dc739ee8d7b23ecb8d2e9"}
Oct 30 21:56:46.552949 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:46.552936    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad56a5184f817ba897593de7a4821990f627dbfee95dc739ee8d7b23ecb8d2e9"
Oct 30 21:56:46.553120 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:46.553107    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b4645f5-mmmfg"
Oct 30 21:56:46.591465 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:46.587476    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:56:46.733370 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:46.728533    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b" podStartSLOduration=1.7285067079999998 podStartE2EDuration="1.728506708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:46.684998243 +0000 UTC m=+1401.159502224" watchObservedRunningTime="2026-10-30 21:56:46.728506708 +0000 UTC m=+1401.203010698"
Oct 30 21:56:46.842016 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:46.841876    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c" podStartSLOduration=2.841854293 podStartE2EDuration="2.841854293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:46.820996294 +0000 UTC m=+1401.295500277" watchObservedRunningTime="2026-10-30 21:56:46.841854293 +0000 UTC m=+1401.316358273"
Oct 30 21:56:47.067372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:47.066705    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb" podStartSLOduration=2.066685099 podStartE2EDuration="2.066685099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:47.064750432 +0000 UTC m=+1401.539254448" watchObservedRunningTime="2026-10-30 21:56:47.066685099 +0000 UTC m=+1401.541189077"
Oct 30 21:56:47.227899 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:47.227753    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:56:47.267388 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:47.234488    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 30 21:56:47.267388 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:47.234545    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c" podUID="c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 30 21:56:47.519954 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:47.519874    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:56:47.520291 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:56:47.520268    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:56:47.550974 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:47.550936    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 21:56:47.551257 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:47.551223    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b" podUID="09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 21:56:47.603374 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:47.587314    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 21:56:47.603374 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:47.587397    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb" podUID="14b7cb87-8d56-4822-9d11-1cd8185c2732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 21:56:47.905544 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:47.905435    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:48.101375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:48.100449    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:48.105372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:48.101628    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:48.105372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:48.101952    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:56:48.125427 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:48.122439    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:48.140213 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:48.139447    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:48.158159 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:48.158063    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:48.172485 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:48.172441    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:48.173064 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:48.173031    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml" podUID="01fdf382-6833-463a-bed9-c7c096166819" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Oct 30 21:56:48.218300 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:48.218267    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 30 21:56:48.218617 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:48.218591    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c" podUID="c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 30 21:56:48.230408 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:48.230299    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml" podUID="01fdf382-6833-463a-bed9-c7c096166819" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:48.269510 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:48.269456    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:48.291486 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:48.291438    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:48.446649 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:48.446510    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:48.504831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:48.504797    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 21:56:48.505547 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:48.505509    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b" podUID="09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 21:56:48.585850 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:48.585809    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 21:56:48.586151 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:48.586122    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb" podUID="14b7cb87-8d56-4822-9d11-1cd8185c2732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 21:56:48.794432 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:48.794387    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": context deadline exceeded"
Oct 30 21:56:48.910010 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:48.909968    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:48.910597 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:48.910576    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:56:49.033884 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.033843    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:49.075562 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.075460    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:49.101507 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.101456    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:49.102712 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.102685    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:56:49.152375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.151476    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:49.152375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.151929    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:56:49.152375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.152218    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Oct 30 21:56:49.219393 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.152689    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:56:49.219393 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.152749    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:56:49.219393 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.183469    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:56:49.219393 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.183942    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc" podUID="f6a9f73b-bb1f-4270-a38d-3703c46fae07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:56:49.219393 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.185449    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 30 21:56:49.219393 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.186277    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:56:49.219393 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.186336    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:56:49.219393 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.187232    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml" podUID="01fdf382-6833-463a-bed9-c7c096166819" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:49.219393 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.205989    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:56:49.219393 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.207743    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:56:49.219393 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.208148    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:56:49.219393 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.208579    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 21:56:49.272498 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.272454    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:56:49.330435 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.303764    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 30 21:56:49.330435 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.303828    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c" podUID="c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 30 21:56:49.393375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.368435    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:56:49.393375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.368812    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:56:49.493385 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.477318    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:56:49.542451 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.531989    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:56:49.596685 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.596568    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 21:56:49.596685 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.596657    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b" podUID="09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 21:56:49.634558 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.634523    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 21:56:49.634844 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.634810    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb" podUID="14b7cb87-8d56-4822-9d11-1cd8185c2732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 21:56:49.822379 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.822319    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": context deadline exceeded"
Oct 30 21:56:49.829500 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.829457    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded"
Oct 30 21:56:49.912528 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:49.911407    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:50.104496 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:50.104446    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:50.155575 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:50.155509    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:56:50.157161 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:50.157078    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:50.163557 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:50.163477    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:56:50.171957 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:50.171922    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:56:50.194898 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:50.194858    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml" podUID="01fdf382-6833-463a-bed9-c7c096166819" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Oct 30 21:56:50.241388 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:50.216698    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:50.241388 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:50.216946    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:56:50.241388 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:50.217096    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc" podUID="f6a9f73b-bb1f-4270-a38d-3703c46fae07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:56:50.241388 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:50.217308    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:56:50.241388 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:50.218295    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:56:50.410397 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:50.389845    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 30 21:56:50.410397 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:50.389911    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c" podUID="c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 30 21:56:50.410397 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:50.390630    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 21:56:50.410884 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:50.410844    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:56:50.704399 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:50.660975    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:50.843416 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:50.818469    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 21:56:50.843416 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:50.818533    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb" podUID="14b7cb87-8d56-4822-9d11-1cd8185c2732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 21:56:50.843416 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:50.819752    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 21:56:50.843416 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:50.819804    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b" podUID="09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 21:56:50.937897 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:50.937836    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:56:51.059639 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:51.043019    5087 patch_prober.go:29] interesting pod/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:56:51.094011 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:51.093964    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc" podUID="dafd13b7-c191-4d83-b94c-371bf53a9fe5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:51.217376 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:51.194451    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:51.226382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:51.219375    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:51.384538 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:51.384428    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": context deadline exceeded"
Oct 30 21:56:51.422375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:51.389606    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:51.565777 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:51.424213    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:56:51.566296 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:51.424285    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:51.566707 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:51.424309    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:51.575383 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:51.434447    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 21:56:51.575383 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:51.436064    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml" podUID="01fdf382-6833-463a-bed9-c7c096166819" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Oct 30 21:56:51.575383 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:51.436314    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 30 21:56:51.579375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:51.436334    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:56:51.579375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:51.436386    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc" podUID="f6a9f73b-bb1f-4270-a38d-3703c46fae07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:56:51.613595 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:51.438104    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 30 21:56:51.613595 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:51.596878    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c" podUID="c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 30 21:56:51.613595 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:51.540106    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 21:56:51.613595 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:51.596939    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b" podUID="09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 21:56:51.635501 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:51.629689    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 21:56:51.635501 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:51.629749    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb" podUID="14b7cb87-8d56-4822-9d11-1cd8185c2732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 21:56:51.673282 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:51.664946    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:52.108467 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:52.101596    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:56:52.181551 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:52.181499    5087 patch_prober.go:29] interesting pod/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:56:52.181902 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:52.181830    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj" podUID="284b12e8-9e86-4406-8494-360e02159b74" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:52.237376 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:52.221438    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:52.237376 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:52.221835    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:56:52.237376 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:52.223375    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 30 21:56:52.237376 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:52.223418    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c" podUID="c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 30 21:56:52.492461 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:52.492290    5087 patch_prober.go:29] interesting pod/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:56:52.492461 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:52.492382    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf" podUID="b2fda28b-f446-4917-bbe1-721ecff56fd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:52.505229 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:52.501503    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 21:56:52.505229 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:52.501556    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b" podUID="09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 21:56:52.581380 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:52.565995    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:52.587302 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:52.581956    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc" podUID="f6a9f73b-bb1f-4270-a38d-3703c46fae07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:52.592373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:52.587565    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:56:52.637447 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:52.592581    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:56:52.637447 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:52.595889    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 21:56:52.637447 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:52.596107    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 21:56:52.637447 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:52.596284    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml" podUID="01fdf382-6833-463a-bed9-c7c096166819" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Oct 30 21:56:52.637447 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:52.596522    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 30 21:56:52.637447 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:52.596689    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:56:52.637447 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:52.614471    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 21:56:52.637447 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:52.614523    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb" podUID="14b7cb87-8d56-4822-9d11-1cd8185c2732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 21:56:52.666672 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:52.665367    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:53.120377 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:53.119745    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:56:53.120865 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:53.120479    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:53.230238 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:53.229022    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 30 21:56:53.230238 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:53.229086    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c" podUID="c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 30 21:56:53.230238 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:53.229167    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:53.231162 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:53.230633    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:56:53.514373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:53.511725    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 21:56:53.514373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:53.511823    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b" podUID="09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 21:56:53.601373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:53.587323    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc" podUID="f6a9f73b-bb1f-4270-a38d-3703c46fae07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:53.601373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:53.588797    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:53.601373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:53.594068    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:56:53.629378 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:53.602739    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:53.632971 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:53.632915    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Oct 30 21:56:53.694871 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:53.694828    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:56:53.722383 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:53.697288    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 21:56:53.871627 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:53.800486    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 21:56:53.871627 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:53.808527    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 30 21:56:53.871627 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:53.846434    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 21:56:53.871627 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:53.852201    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb" podUID="14b7cb87-8d56-4822-9d11-1cd8185c2732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 21:56:53.871627 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:53.850484    5087 patch_prober.go:29] interesting pod/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:56:53.871627 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:53.852264    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429" podUID="86ab6a2a-10d7-4957-96ac-4cc29e0fb409" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:53.901595 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:53.901548    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml" podUID="01fdf382-6833-463a-bed9-c7c096166819" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:54.133251 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.133149    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:56:54.257384 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.242022    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:54.289219 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.243553    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:54.293379 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.243921    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 30 21:56:54.293725 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.293669    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c" podUID="c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 30 21:56:54.506937 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.500805    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 21:56:54.506937 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.500891    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b" podUID="09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 21:56:54.545408 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.523341    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18fa312c-jk4nh"]
Oct 30 21:56:54.546635 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.546610    5087 cpu_manager.go:414] "containerMap: removing container" podUID="40d91704-2d27-4030-b79e-74046213e9d6" containerName="revoke-keys"
Oct 30 21:56:54.546791 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.546776    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40d91704-2d27-4030-b79e-74046213e9d6" containerName="revoke-keys"
Oct 30 21:56:54.547084 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.547068    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="40d91704-2d27-4030-b79e-74046213e9d6" containerName="revoke-keys"
Oct 30 21:56:54.572243 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.572214    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:56:54.572485 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.572469    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18fa312c-jk4nh"]
Oct 30 21:56:54.595987 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.595960    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fa312c-jk4nh"
Oct 30 21:56:54.604884 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.604864    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:56:54.697368 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.682716    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:56:54.697368 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.682793    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:56:54.697368 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.692946    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:54.697368 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.693265    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:56:54.697368 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.693549    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:56:54.697368 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.693784    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:56:54.723231 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.721632    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 21:56:54.723231 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.721708    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb" podUID="14b7cb87-8d56-4822-9d11-1cd8185c2732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 21:56:54.737435 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.737165    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:56:54.753867 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.752063    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbc8w\" (UniqueName: \"kubernetes.io/projected/f9b38122-a71b-4f4e-b70d-e03ad0a69b21-kube-api-access-kbc8w\") pod \"maas-api-revoke-sub-18fa312c-jk4nh\" (UID: \"f9b38122-a71b-4f4e-b70d-e03ad0a69b21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fa312c-jk4nh"
Oct 30 21:56:54.753867 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.752442    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9b38122-a71b-4f4e-b70d-e03ad0a69b21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18fa312c-jk4nh\" (UID: \"f9b38122-a71b-4f4e-b70d-e03ad0a69b21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fa312c-jk4nh"
Oct 30 21:56:54.753867 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.752521    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9b38122-a71b-4f4e-b70d-e03ad0a69b21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18fa312c-jk4nh\" (UID: \"f9b38122-a71b-4f4e-b70d-e03ad0a69b21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fa312c-jk4nh"
Oct 30 21:56:54.911241 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.908440    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 21:56:54.911241 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.908727    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 30 21:56:54.911241 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.909771    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9b38122-a71b-4f4e-b70d-e03ad0a69b21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18fa312c-jk4nh\" (UID: \"f9b38122-a71b-4f4e-b70d-e03ad0a69b21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fa312c-jk4nh"
Oct 30 21:56:54.911241 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.909917    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbc8w\" (UniqueName: \"kubernetes.io/projected/f9b38122-a71b-4f4e-b70d-e03ad0a69b21-kube-api-access-kbc8w\") pod \"maas-api-revoke-sub-18fa312c-jk4nh\" (UID: \"f9b38122-a71b-4f4e-b70d-e03ad0a69b21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fa312c-jk4nh"
Oct 30 21:56:54.911241 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.910169    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9b38122-a71b-4f4e-b70d-e03ad0a69b21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18fa312c-jk4nh\" (UID: \"f9b38122-a71b-4f4e-b70d-e03ad0a69b21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fa312c-jk4nh"
Oct 30 21:56:54.911241 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.911028    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9b38122-a71b-4f4e-b70d-e03ad0a69b21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-18fa312c-jk4nh\" (UID: \"f9b38122-a71b-4f4e-b70d-e03ad0a69b21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fa312c-jk4nh"
Oct 30 21:56:54.936251 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.934001    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:56:54.961370 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.954031    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:54.961370 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:54.957995    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9b38122-a71b-4f4e-b70d-e03ad0a69b21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-18fa312c-jk4nh\" (UID: \"f9b38122-a71b-4f4e-b70d-e03ad0a69b21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fa312c-jk4nh"
Oct 30 21:56:55.050764 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:55.050734    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:56:55.160515 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:55.135214    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbc8w\" (UniqueName: \"kubernetes.io/projected/f9b38122-a71b-4f4e-b70d-e03ad0a69b21-kube-api-access-kbc8w\") pod \"maas-api-revoke-sub-18fa312c-jk4nh\" (UID: \"f9b38122-a71b-4f4e-b70d-e03ad0a69b21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fa312c-jk4nh"
Oct 30 21:56:55.222560 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:55.222468    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 30 21:56:55.223454 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:55.223413    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c" podUID="c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 30 21:56:55.345506 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:55.345460    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:56:55.375422 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:55.375201    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fa312c-jk4nh"
Oct 30 21:56:55.440546 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:55.422454    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:55.505525 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:55.505493    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 21:56:55.615445 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:55.513436    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b" podUID="09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 21:56:55.615445 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:55.609796    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 21:56:55.615445 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:55.609840    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb" podUID="14b7cb87-8d56-4822-9d11-1cd8185c2732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 21:56:55.615445 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:55.609918    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc" podUID="f6a9f73b-bb1f-4270-a38d-3703c46fae07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:55.687638 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:55.687595    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:55.779024 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:55.730888    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:55.779865 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:55.730926    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Oct 30 21:56:55.805408 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:55.730977    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:56:55.947380 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:55.946779    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:56.035155 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:56.030421    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml" podUID="01fdf382-6833-463a-bed9-c7c096166819" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Oct 30 21:56:56.154383 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:56.153863    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:56.154383 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:56.153948    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 21:56:56.154383 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:56.153992    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:56:56.393478 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:56.366453    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:56.394309 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:56.391306    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 30 21:56:56.394521 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:56.394493    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c" podUID="c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 30 21:56:56.426751 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:56.426711    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:56.515458 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:56.515422    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 21:56:56.517164 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:56.517131    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b" podUID="09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 21:56:56.614793 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:56.614745    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc" podUID="f6a9f73b-bb1f-4270-a38d-3703c46fae07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:56:56.635695 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:56.635654    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:56.667203 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:56.646270    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 21:56:56.692997 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:56.692947    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb" podUID="14b7cb87-8d56-4822-9d11-1cd8185c2732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 21:56:56.693766 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:56.693740    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:56:56.736020 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:56.735974    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:56.807502 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:56.807451    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:56.808177 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:56.808141    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:56.808694 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:56.808669    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:56.965435 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:56.965008    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:57.076373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:57.036714    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml" podUID="01fdf382-6833-463a-bed9-c7c096166819" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:57.129731 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:57.104673    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:56:57.130376 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:57.076176    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc" podUID="f6a9f73b-bb1f-4270-a38d-3703c46fae07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:57.131019 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:57.076238    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:57.153184 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:57.153142    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:57.153808 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:57.153788    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:56:57.365369 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:57.365226    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 30 21:56:57.365369 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:57.365289    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c" podUID="c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 30 21:56:57.397374 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:57.396233    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:57.397374 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:57.397103    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:56:57.397940 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:57.397831    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:56:57.415369 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:57.413183    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 21:56:57.456376 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:57.452440    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:57.514277 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:57.514244    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 21:56:57.514596 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:57.514532    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b" podUID="09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 21:56:57.601765 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:57.597787    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 21:56:57.601765 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:57.597853    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb" podUID="14b7cb87-8d56-4822-9d11-1cd8185c2732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 21:56:57.634650 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:57.632594    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc" podUID="f6a9f73b-bb1f-4270-a38d-3703c46fae07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:57.833883 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:57.833820    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:56:57.836061 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:57.834250    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:56:58.024388 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:58.022324    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:58.026965 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:58.026578    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:56:58.129444 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:58.127531    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:56:58.220427 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:58.218852    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 30 21:56:58.220427 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:58.218914    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c" podUID="c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 30 21:56:58.329391 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:58.329339    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:56:58.329659 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:58.329644    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:56:58.329866 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:58.329850    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:56:58.330009 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:58.329997    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:56:58.330135 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:58.330123    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:56:58.330874 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:58.330843    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:56:58.332082 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:58.332065    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:56:58.332309 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:58.332295    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:56:58.332735 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:58.332713    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:56:58.340375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:58.335955    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:56:58.419783 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:58.414537    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:56:58.427040 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:58.427014    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:56:58.518285 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:58.518073    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 21:56:58.518608 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:58.518575    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b" podUID="09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 21:56:58.627781 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:58.625496    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 21:56:58.627781 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:58.625554    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb" podUID="14b7cb87-8d56-4822-9d11-1cd8185c2732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 21:56:58.785452 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:58.785309    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18fa312c-jk4nh"]
Oct 30 21:56:59.236646 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:59.235377    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 30 21:56:59.236646 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:59.235432    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c" podUID="c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 30 21:56:59.321153 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:59.321076    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fa312c-jk4nh" event={"ID":"f9b38122-a71b-4f4e-b70d-e03ad0a69b21","Type":"ContainerStarted","Data":"18bdb90462162dfc81f82e86ed9bc7649d3474bf83fb7e712ff8088d01eb3c73"}
Oct 30 21:56:59.511752 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:59.511719    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Oct 30 21:56:59.512188 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:59.511788    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b" podUID="09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Oct 30 21:56:59.585832 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:59.585799    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 30 21:56:59.586233 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:56:59.586207    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb" podUID="14b7cb87-8d56-4822-9d11-1cd8185c2732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 30 21:57:00.217260 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:00.216052    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:57:00.218164 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:00.218129    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 30 21:57:00.218295 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:00.218188    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c" podUID="c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 30 21:57:00.392112 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:00.392074    5087 generic.go:356] "Generic (PLEG): container finished" podID="f9b38122-a71b-4f4e-b70d-e03ad0a69b21" containerID="ffc8e2e96735b708913ea06ab0648e40044d04ca863f27524207ac3fab09f10d" exitCode=0
Oct 30 21:57:00.392308 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:00.392159    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fa312c-jk4nh" event={"ID":"f9b38122-a71b-4f4e-b70d-e03ad0a69b21","Type":"ContainerDied","Data":"ffc8e2e96735b708913ea06ab0648e40044d04ca863f27524207ac3fab09f10d"}
Oct 30 21:57:00.497830 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:00.497794    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:57:00.584610 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:00.584576    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:57:00.700622 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:00.700592    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:57:00.869031 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:00.868933    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:57:01.220324 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:01.220238    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:57:01.221928 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:01.221900    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 21:57:01.401865 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:01.401564    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 21:57:01.401865 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:01.401624    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 21:57:01.634080 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:01.633959    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fa312c-jk4nh"
Oct 30 21:57:01.702943 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:01.702852    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9b38122-a71b-4f4e-b70d-e03ad0a69b21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9b38122-a71b-4f4e-b70d-e03ad0a69b21-maas-api-openshift-ca\") pod \"f9b38122-a71b-4f4e-b70d-e03ad0a69b21\" (UID: \"f9b38122-a71b-4f4e-b70d-e03ad0a69b21\") "
Oct 30 21:57:01.703432 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:01.703406    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9b38122-a71b-4f4e-b70d-e03ad0a69b21-kube-api-access-kbc8w\" (UniqueName: \"kubernetes.io/projected/f9b38122-a71b-4f4e-b70d-e03ad0a69b21-kube-api-access-kbc8w\") pod \"f9b38122-a71b-4f4e-b70d-e03ad0a69b21\" (UID: \"f9b38122-a71b-4f4e-b70d-e03ad0a69b21\") "
Oct 30 21:57:01.703661 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:01.703642    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9b38122-a71b-4f4e-b70d-e03ad0a69b21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9b38122-a71b-4f4e-b70d-e03ad0a69b21-maas-api-xks-ca\") pod \"f9b38122-a71b-4f4e-b70d-e03ad0a69b21\" (UID: \"f9b38122-a71b-4f4e-b70d-e03ad0a69b21\") "
Oct 30 21:57:01.703953 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:01.703928    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b38122-a71b-4f4e-b70d-e03ad0a69b21-maas-api-openshift-ca" pod "f9b38122-a71b-4f4e-b70d-e03ad0a69b21" (UID: "f9b38122-a71b-4f4e-b70d-e03ad0a69b21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:01.704975 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:01.704947    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9b38122-a71b-4f4e-b70d-e03ad0a69b21-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:57:01.707905 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:01.707800    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b38122-a71b-4f4e-b70d-e03ad0a69b21-kube-api-access-kbc8w" pod "f9b38122-a71b-4f4e-b70d-e03ad0a69b21" (UID: "f9b38122-a71b-4f4e-b70d-e03ad0a69b21"). InnerVolumeSpecName "kube-api-access-kbc8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:01.709972 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:01.709906    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9b38122-a71b-4f4e-b70d-e03ad0a69b21-maas-api-xks-ca" pod "f9b38122-a71b-4f4e-b70d-e03ad0a69b21" (UID: "f9b38122-a71b-4f4e-b70d-e03ad0a69b21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:01.805922 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:01.805893    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbc8w\" (UniqueName: \"kubernetes.io/projected/f9b38122-a71b-4f4e-b70d-e03ad0a69b21-kube-api-access-kbc8w\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:57:01.805922 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:01.805920    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9b38122-a71b-4f4e-b70d-e03ad0a69b21-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:57:02.412221 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:02.412187    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fa312c-jk4nh"
Oct 30 21:57:02.412221 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:02.412199    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-18fa312c-jk4nh" event={"ID":"f9b38122-a71b-4f4e-b70d-e03ad0a69b21","Type":"ContainerDied","Data":"18bdb90462162dfc81f82e86ed9bc7649d3474bf83fb7e712ff8088d01eb3c73"}
Oct 30 21:57:02.412521 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:02.412247    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18bdb90462162dfc81f82e86ed9bc7649d3474bf83fb7e712ff8088d01eb3c73"
Oct 30 21:57:04.459778 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:04.459744    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98324f6c-5n98h"]
Oct 30 21:57:04.460458 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:04.460437    5087 cpu_manager.go:414] "containerMap: removing container" podUID="f9b38122-a71b-4f4e-b70d-e03ad0a69b21" containerName="revoke-keys"
Oct 30 21:57:04.460458 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:04.460457    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9b38122-a71b-4f4e-b70d-e03ad0a69b21" containerName="revoke-keys"
Oct 30 21:57:04.460641 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:04.460582    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9b38122-a71b-4f4e-b70d-e03ad0a69b21" containerName="revoke-keys"
Oct 30 21:57:04.464575 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:04.464554    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98324f6c-5n98h"
Oct 30 21:57:04.516778 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:04.516714    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:57:04.517331 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:04.516875    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:57:04.520803 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:04.520759    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98324f6c-5n98h"]
Oct 30 21:57:04.548051 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:04.548024    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62db9532-ad32-4f35-85dd-0ad224192a66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98324f6c-5n98h\" (UID: \"62db9532-ad32-4f35-85dd-0ad224192a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98324f6c-5n98h"
Oct 30 21:57:04.548218 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:04.548198    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62db9532-ad32-4f35-85dd-0ad224192a66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98324f6c-5n98h\" (UID: \"62db9532-ad32-4f35-85dd-0ad224192a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98324f6c-5n98h"
Oct 30 21:57:04.548309 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:04.548294    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnt7h\" (UniqueName: \"kubernetes.io/projected/62db9532-ad32-4f35-85dd-0ad224192a66-kube-api-access-tnt7h\") pod \"maas-api-revoke-sub-98324f6c-5n98h\" (UID: \"62db9532-ad32-4f35-85dd-0ad224192a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98324f6c-5n98h"
Oct 30 21:57:04.650078 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:04.650047    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62db9532-ad32-4f35-85dd-0ad224192a66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98324f6c-5n98h\" (UID: \"62db9532-ad32-4f35-85dd-0ad224192a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98324f6c-5n98h"
Oct 30 21:57:04.650247 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:04.650126    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnt7h\" (UniqueName: \"kubernetes.io/projected/62db9532-ad32-4f35-85dd-0ad224192a66-kube-api-access-tnt7h\") pod \"maas-api-revoke-sub-98324f6c-5n98h\" (UID: \"62db9532-ad32-4f35-85dd-0ad224192a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98324f6c-5n98h"
Oct 30 21:57:04.650247 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:04.650232    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62db9532-ad32-4f35-85dd-0ad224192a66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98324f6c-5n98h\" (UID: \"62db9532-ad32-4f35-85dd-0ad224192a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98324f6c-5n98h"
Oct 30 21:57:04.650873 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:04.650847    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62db9532-ad32-4f35-85dd-0ad224192a66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98324f6c-5n98h\" (UID: \"62db9532-ad32-4f35-85dd-0ad224192a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98324f6c-5n98h"
Oct 30 21:57:04.652372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:04.652332    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62db9532-ad32-4f35-85dd-0ad224192a66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98324f6c-5n98h\" (UID: \"62db9532-ad32-4f35-85dd-0ad224192a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98324f6c-5n98h"
Oct 30 21:57:04.708480 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:04.708456    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnt7h\" (UniqueName: \"kubernetes.io/projected/62db9532-ad32-4f35-85dd-0ad224192a66-kube-api-access-tnt7h\") pod \"maas-api-revoke-sub-98324f6c-5n98h\" (UID: \"62db9532-ad32-4f35-85dd-0ad224192a66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98324f6c-5n98h"
Oct 30 21:57:04.782106 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:04.782079    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98324f6c-5n98h"
Oct 30 21:57:05.163649 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:57:05.163621    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62db9532_ad32_4f35_85dd_0ad224192a66.slice/crio-201e908bcee4d3e0d391ddfda95b0fdfedbe3739b342f9d0e5d0114acb4f6c27 WatchSource:0}: Error finding container 201e908bcee4d3e0d391ddfda95b0fdfedbe3739b342f9d0e5d0114acb4f6c27: Status 404 returned error can't find the container with id 201e908bcee4d3e0d391ddfda95b0fdfedbe3739b342f9d0e5d0114acb4f6c27
Oct 30 21:57:05.165524 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:05.165499    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98324f6c-5n98h"]
Oct 30 21:57:05.457783 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:05.455089    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98324f6c-5n98h" event={"ID":"62db9532-ad32-4f35-85dd-0ad224192a66","Type":"ContainerStarted","Data":"201e908bcee4d3e0d391ddfda95b0fdfedbe3739b342f9d0e5d0114acb4f6c27"}
Oct 30 21:57:06.464721 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:06.464642    5087 generic.go:356] "Generic (PLEG): container finished" podID="62db9532-ad32-4f35-85dd-0ad224192a66" containerID="09da43b8edf3e0b15768d798b13b9a75652be1ba6c40a8220bc6db6e6ca40e77" exitCode=0
Oct 30 21:57:06.465076 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:06.464723    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98324f6c-5n98h" event={"ID":"62db9532-ad32-4f35-85dd-0ad224192a66","Type":"ContainerDied","Data":"09da43b8edf3e0b15768d798b13b9a75652be1ba6c40a8220bc6db6e6ca40e77"}
Oct 30 21:57:07.666795 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:07.666763    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98324f6c-5n98h"
Oct 30 21:57:07.791770 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:07.791739    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62db9532-ad32-4f35-85dd-0ad224192a66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62db9532-ad32-4f35-85dd-0ad224192a66-maas-api-xks-ca\") pod \"62db9532-ad32-4f35-85dd-0ad224192a66\" (UID: \"62db9532-ad32-4f35-85dd-0ad224192a66\") "
Oct 30 21:57:07.791935 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:07.791870    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62db9532-ad32-4f35-85dd-0ad224192a66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62db9532-ad32-4f35-85dd-0ad224192a66-maas-api-openshift-ca\") pod \"62db9532-ad32-4f35-85dd-0ad224192a66\" (UID: \"62db9532-ad32-4f35-85dd-0ad224192a66\") "
Oct 30 21:57:07.792031 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:07.792013    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62db9532-ad32-4f35-85dd-0ad224192a66-kube-api-access-tnt7h\" (UniqueName: \"kubernetes.io/projected/62db9532-ad32-4f35-85dd-0ad224192a66-kube-api-access-tnt7h\") pod \"62db9532-ad32-4f35-85dd-0ad224192a66\" (UID: \"62db9532-ad32-4f35-85dd-0ad224192a66\") "
Oct 30 21:57:07.792265 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:07.792236    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62db9532-ad32-4f35-85dd-0ad224192a66-maas-api-openshift-ca" pod "62db9532-ad32-4f35-85dd-0ad224192a66" (UID: "62db9532-ad32-4f35-85dd-0ad224192a66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:07.792727 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:07.792694    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62db9532-ad32-4f35-85dd-0ad224192a66-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:57:07.794140 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:07.794117    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62db9532-ad32-4f35-85dd-0ad224192a66-maas-api-xks-ca" pod "62db9532-ad32-4f35-85dd-0ad224192a66" (UID: "62db9532-ad32-4f35-85dd-0ad224192a66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:07.794234 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:07.794137    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62db9532-ad32-4f35-85dd-0ad224192a66-kube-api-access-tnt7h" pod "62db9532-ad32-4f35-85dd-0ad224192a66" (UID: "62db9532-ad32-4f35-85dd-0ad224192a66"). InnerVolumeSpecName "kube-api-access-tnt7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:07.894196 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:07.894161    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnt7h\" (UniqueName: \"kubernetes.io/projected/62db9532-ad32-4f35-85dd-0ad224192a66-kube-api-access-tnt7h\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:57:07.894196 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:07.894193    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62db9532-ad32-4f35-85dd-0ad224192a66-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:57:08.482218 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:08.482190    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98324f6c-5n98h"
Oct 30 21:57:08.482413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:08.482226    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98324f6c-5n98h" event={"ID":"62db9532-ad32-4f35-85dd-0ad224192a66","Type":"ContainerDied","Data":"201e908bcee4d3e0d391ddfda95b0fdfedbe3739b342f9d0e5d0114acb4f6c27"}
Oct 30 21:57:08.482413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:08.482308    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="201e908bcee4d3e0d391ddfda95b0fdfedbe3739b342f9d0e5d0114acb4f6c27"
Oct 30 21:57:17.135872 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:17.135840    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fbb8fca-dmqnp"]
Oct 30 21:57:17.137719 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:17.137691    5087 cpu_manager.go:414] "containerMap: removing container" podUID="62db9532-ad32-4f35-85dd-0ad224192a66" containerName="revoke-keys"
Oct 30 21:57:17.137719 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:17.137716    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62db9532-ad32-4f35-85dd-0ad224192a66" containerName="revoke-keys"
Oct 30 21:57:17.137917 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:17.137850    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="62db9532-ad32-4f35-85dd-0ad224192a66" containerName="revoke-keys"
Oct 30 21:57:17.141688 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:17.141662    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fbb8fca-dmqnp"
Oct 30 21:57:17.171985 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:57:17.171947    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:57:17.195326 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:17.195302    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:57:17.226382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:17.226320    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4b8006f-127c-4a1b-8019-e0c5176e8d75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fbb8fca-dmqnp\" (UID: \"d4b8006f-127c-4a1b-8019-e0c5176e8d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fbb8fca-dmqnp"
Oct 30 21:57:17.226522 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:17.226491    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjwmq\" (UniqueName: \"kubernetes.io/projected/d4b8006f-127c-4a1b-8019-e0c5176e8d75-kube-api-access-sjwmq\") pod \"maas-api-revoke-sub-5fbb8fca-dmqnp\" (UID: \"d4b8006f-127c-4a1b-8019-e0c5176e8d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fbb8fca-dmqnp"
Oct 30 21:57:17.226564 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:17.226545    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4b8006f-127c-4a1b-8019-e0c5176e8d75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fbb8fca-dmqnp\" (UID: \"d4b8006f-127c-4a1b-8019-e0c5176e8d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fbb8fca-dmqnp"
Oct 30 21:57:17.248409 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:17.245243    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:57:17.251771 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:17.251742    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fbb8fca-dmqnp"]
Oct 30 21:57:17.327457 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:17.327426    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4b8006f-127c-4a1b-8019-e0c5176e8d75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fbb8fca-dmqnp\" (UID: \"d4b8006f-127c-4a1b-8019-e0c5176e8d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fbb8fca-dmqnp"
Oct 30 21:57:17.327615 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:17.327488    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjwmq\" (UniqueName: \"kubernetes.io/projected/d4b8006f-127c-4a1b-8019-e0c5176e8d75-kube-api-access-sjwmq\") pod \"maas-api-revoke-sub-5fbb8fca-dmqnp\" (UID: \"d4b8006f-127c-4a1b-8019-e0c5176e8d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fbb8fca-dmqnp"
Oct 30 21:57:17.327615 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:17.327520    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4b8006f-127c-4a1b-8019-e0c5176e8d75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fbb8fca-dmqnp\" (UID: \"d4b8006f-127c-4a1b-8019-e0c5176e8d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fbb8fca-dmqnp"
Oct 30 21:57:17.328250 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:17.328195    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4b8006f-127c-4a1b-8019-e0c5176e8d75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fbb8fca-dmqnp\" (UID: \"d4b8006f-127c-4a1b-8019-e0c5176e8d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fbb8fca-dmqnp"
Oct 30 21:57:17.330168 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:17.330149    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4b8006f-127c-4a1b-8019-e0c5176e8d75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fbb8fca-dmqnp\" (UID: \"d4b8006f-127c-4a1b-8019-e0c5176e8d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fbb8fca-dmqnp"
Oct 30 21:57:17.389139 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:17.389067    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjwmq\" (UniqueName: \"kubernetes.io/projected/d4b8006f-127c-4a1b-8019-e0c5176e8d75-kube-api-access-sjwmq\") pod \"maas-api-revoke-sub-5fbb8fca-dmqnp\" (UID: \"d4b8006f-127c-4a1b-8019-e0c5176e8d75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fbb8fca-dmqnp"
Oct 30 21:57:17.464109 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:17.464087    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fbb8fca-dmqnp"
Oct 30 21:57:17.849103 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:57:17.849068    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b8006f_127c_4a1b_8019_e0c5176e8d75.slice/crio-5c06c68b608bd1ec612b2c1cb199cbbd676bd70be2fe7a00f23bef4d06e56413 WatchSource:0}: Error finding container 5c06c68b608bd1ec612b2c1cb199cbbd676bd70be2fe7a00f23bef4d06e56413: Status 404 returned error can't find the container with id 5c06c68b608bd1ec612b2c1cb199cbbd676bd70be2fe7a00f23bef4d06e56413
Oct 30 21:57:17.851325 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:17.851304    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fbb8fca-dmqnp"]
Oct 30 21:57:18.575007 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:18.574968    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fbb8fca-dmqnp" event={"ID":"d4b8006f-127c-4a1b-8019-e0c5176e8d75","Type":"ContainerStarted","Data":"111995d0d829bc5d38de856ca6b3fab1ce7163d08035600ce8df2a7d7c13d8f3"}
Oct 30 21:57:18.575472 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:18.575038    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fbb8fca-dmqnp" event={"ID":"d4b8006f-127c-4a1b-8019-e0c5176e8d75","Type":"ContainerStarted","Data":"5c06c68b608bd1ec612b2c1cb199cbbd676bd70be2fe7a00f23bef4d06e56413"}
Oct 30 21:57:18.696748 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:18.696692    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fbb8fca-dmqnp" podStartSLOduration=1.6966733330000001 podStartE2EDuration="1.696673333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:18.689137196 +0000 UTC m=+1433.163641178" watchObservedRunningTime="2026-10-30 21:57:18.696673333 +0000 UTC m=+1433.171177312"
Oct 30 21:57:19.592560 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:19.592519    5087 generic.go:356] "Generic (PLEG): container finished" podID="d4b8006f-127c-4a1b-8019-e0c5176e8d75" containerID="111995d0d829bc5d38de856ca6b3fab1ce7163d08035600ce8df2a7d7c13d8f3" exitCode=0
Oct 30 21:57:19.593798 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:19.593285    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fbb8fca-dmqnp" event={"ID":"d4b8006f-127c-4a1b-8019-e0c5176e8d75","Type":"ContainerDied","Data":"111995d0d829bc5d38de856ca6b3fab1ce7163d08035600ce8df2a7d7c13d8f3"}
Oct 30 21:57:20.797596 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:20.797569    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fbb8fca-dmqnp"
Oct 30 21:57:20.929596 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:20.929523    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4b8006f-127c-4a1b-8019-e0c5176e8d75-kube-api-access-sjwmq\" (UniqueName: \"kubernetes.io/projected/d4b8006f-127c-4a1b-8019-e0c5176e8d75-kube-api-access-sjwmq\") pod \"d4b8006f-127c-4a1b-8019-e0c5176e8d75\" (UID: \"d4b8006f-127c-4a1b-8019-e0c5176e8d75\") "
Oct 30 21:57:20.929770 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:20.929757    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4b8006f-127c-4a1b-8019-e0c5176e8d75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4b8006f-127c-4a1b-8019-e0c5176e8d75-maas-api-xks-ca\") pod \"d4b8006f-127c-4a1b-8019-e0c5176e8d75\" (UID: \"d4b8006f-127c-4a1b-8019-e0c5176e8d75\") "
Oct 30 21:57:20.929845 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:20.929808    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4b8006f-127c-4a1b-8019-e0c5176e8d75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4b8006f-127c-4a1b-8019-e0c5176e8d75-maas-api-openshift-ca\") pod \"d4b8006f-127c-4a1b-8019-e0c5176e8d75\" (UID: \"d4b8006f-127c-4a1b-8019-e0c5176e8d75\") "
Oct 30 21:57:20.930672 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:20.930649    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4b8006f-127c-4a1b-8019-e0c5176e8d75-maas-api-openshift-ca" pod "d4b8006f-127c-4a1b-8019-e0c5176e8d75" (UID: "d4b8006f-127c-4a1b-8019-e0c5176e8d75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:20.932520 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:20.932496    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4b8006f-127c-4a1b-8019-e0c5176e8d75-maas-api-xks-ca" pod "d4b8006f-127c-4a1b-8019-e0c5176e8d75" (UID: "d4b8006f-127c-4a1b-8019-e0c5176e8d75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:20.932626 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:20.932526    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b8006f-127c-4a1b-8019-e0c5176e8d75-kube-api-access-sjwmq" pod "d4b8006f-127c-4a1b-8019-e0c5176e8d75" (UID: "d4b8006f-127c-4a1b-8019-e0c5176e8d75"). InnerVolumeSpecName "kube-api-access-sjwmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:21.031675 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:21.031639    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4b8006f-127c-4a1b-8019-e0c5176e8d75-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:57:21.031675 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:21.031667    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4b8006f-127c-4a1b-8019-e0c5176e8d75-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:57:21.031675 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:21.031678    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjwmq\" (UniqueName: \"kubernetes.io/projected/d4b8006f-127c-4a1b-8019-e0c5176e8d75-kube-api-access-sjwmq\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:57:21.112511 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:21.112326    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f988e89a-g765q"]
Oct 30 21:57:21.113306 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:21.113281    5087 cpu_manager.go:414] "containerMap: removing container" podUID="d4b8006f-127c-4a1b-8019-e0c5176e8d75" containerName="revoke-keys"
Oct 30 21:57:21.113434 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:21.113310    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4b8006f-127c-4a1b-8019-e0c5176e8d75" containerName="revoke-keys"
Oct 30 21:57:21.113563 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:21.113548    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4b8006f-127c-4a1b-8019-e0c5176e8d75" containerName="revoke-keys"
Oct 30 21:57:21.119831 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:21.119808    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988e89a-g765q"
Oct 30 21:57:21.179380 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:21.173642    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f988e89a-g765q"]
Oct 30 21:57:21.250080 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:21.250041    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27kht\" (UniqueName: \"kubernetes.io/projected/b8d32e3b-810b-40d4-a5bd-73b119077b30-kube-api-access-27kht\") pod \"maas-api-revoke-sub-f988e89a-g765q\" (UID: \"b8d32e3b-810b-40d4-a5bd-73b119077b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988e89a-g765q"
Oct 30 21:57:21.250259 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:21.250236    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d32e3b-810b-40d4-a5bd-73b119077b30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f988e89a-g765q\" (UID: \"b8d32e3b-810b-40d4-a5bd-73b119077b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988e89a-g765q"
Oct 30 21:57:21.250454 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:21.250426    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d32e3b-810b-40d4-a5bd-73b119077b30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f988e89a-g765q\" (UID: \"b8d32e3b-810b-40d4-a5bd-73b119077b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988e89a-g765q"
Oct 30 21:57:21.351679 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:21.351646    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27kht\" (UniqueName: \"kubernetes.io/projected/b8d32e3b-810b-40d4-a5bd-73b119077b30-kube-api-access-27kht\") pod \"maas-api-revoke-sub-f988e89a-g765q\" (UID: \"b8d32e3b-810b-40d4-a5bd-73b119077b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988e89a-g765q"
Oct 30 21:57:21.351870 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:21.351744    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d32e3b-810b-40d4-a5bd-73b119077b30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f988e89a-g765q\" (UID: \"b8d32e3b-810b-40d4-a5bd-73b119077b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988e89a-g765q"
Oct 30 21:57:21.351870 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:21.351844    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d32e3b-810b-40d4-a5bd-73b119077b30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f988e89a-g765q\" (UID: \"b8d32e3b-810b-40d4-a5bd-73b119077b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988e89a-g765q"
Oct 30 21:57:21.352610 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:21.352579    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d32e3b-810b-40d4-a5bd-73b119077b30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f988e89a-g765q\" (UID: \"b8d32e3b-810b-40d4-a5bd-73b119077b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988e89a-g765q"
Oct 30 21:57:21.354446 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:21.354423    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d32e3b-810b-40d4-a5bd-73b119077b30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f988e89a-g765q\" (UID: \"b8d32e3b-810b-40d4-a5bd-73b119077b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988e89a-g765q"
Oct 30 21:57:21.409592 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:21.409556    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27kht\" (UniqueName: \"kubernetes.io/projected/b8d32e3b-810b-40d4-a5bd-73b119077b30-kube-api-access-27kht\") pod \"maas-api-revoke-sub-f988e89a-g765q\" (UID: \"b8d32e3b-810b-40d4-a5bd-73b119077b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988e89a-g765q"
Oct 30 21:57:21.464372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:21.460630    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988e89a-g765q"
Oct 30 21:57:21.805421 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:21.788024    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fbb8fca-dmqnp"
Oct 30 21:57:21.842339 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:21.831832    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fbb8fca-dmqnp" event={"ID":"d4b8006f-127c-4a1b-8019-e0c5176e8d75","Type":"ContainerDied","Data":"5c06c68b608bd1ec612b2c1cb199cbbd676bd70be2fe7a00f23bef4d06e56413"}
Oct 30 21:57:21.842724 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:21.842696    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c06c68b608bd1ec612b2c1cb199cbbd676bd70be2fe7a00f23bef4d06e56413"
Oct 30 21:57:21.961404 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:21.956296    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f988e89a-g765q"]
Oct 30 21:57:22.949051 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:22.940217    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988e89a-g765q" event={"ID":"b8d32e3b-810b-40d4-a5bd-73b119077b30","Type":"ContainerStarted","Data":"3cc2b7696bcb0bdbac430124ba2839eed086977ff8493d49cbe05d0148966b19"}
Oct 30 21:57:23.418501 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:23.417454    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:57:23.439504 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:23.439450    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml" podUID="01fdf382-6833-463a-bed9-c7c096166819" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:23.609223 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:23.608461    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 30 21:57:23.675419 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:23.643444    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:23.675419 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:23.643816    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:57:23.830374 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:23.826438    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:57:23.830374 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:23.826521    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx" podUID="3f00bbc1-19c4-447a-9459-3443bbabb845" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:24.197430 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:24.181453    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:24.238399 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:24.220507    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:24.238399 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:24.226684    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:57:24.238399 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:24.227052    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:57:24.247016 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:24.246609    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc" podUID="f6a9f73b-bb1f-4270-a38d-3703c46fae07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:24.383372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:24.382460    5087 generic.go:356] "Generic (PLEG): container finished" podID="b8d32e3b-810b-40d4-a5bd-73b119077b30" containerID="8ee918ffae3ad632371ba5bd9ede01592baa975edaa84dba811a86498f782acd" exitCode=0
Oct 30 21:57:24.383372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:24.382788    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988e89a-g765q" event={"ID":"b8d32e3b-810b-40d4-a5bd-73b119077b30","Type":"ContainerDied","Data":"8ee918ffae3ad632371ba5bd9ede01592baa975edaa84dba811a86498f782acd"}
Oct 30 21:57:24.434373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:24.434170    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:24.438372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:24.434617    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:24.450166 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:24.442064    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml" podUID="01fdf382-6833-463a-bed9-c7c096166819" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:24.457772 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:24.454497    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:57:24.458036 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:24.458017    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:57:24.618376 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:24.608653    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:24.618376 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:24.609982    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:57:24.721915 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:24.721819    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:24.722633 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:24.722609    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:57:24.745671 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:24.723692    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:57:24.747038 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:24.727434    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:57:24.748312 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:24.748289    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:57:25.094372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:25.090442    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:57:25.106375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:25.102422    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5" podUID="9db8cae8-a410-4c25-bdec-cee65fb1a8ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:25.153375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:25.152497    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:25.239492 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:25.183265    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:25.239492 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:25.183717    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:57:25.239492 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:25.228611    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:25.269381 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:25.244913    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:57:25.441537 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:25.251740    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc" podUID="f6a9f73b-bb1f-4270-a38d-3703c46fae07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:57:25.456779 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:25.456754    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:57:25.462377 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:25.457838    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml" podUID="01fdf382-6833-463a-bed9-c7c096166819" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Oct 30 21:57:25.467738 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:25.455506    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:57:25.468172 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:25.312192    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:57:25.469871 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:25.312227    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:57:25.843855 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:25.835127    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Oct 30 21:57:25.843855 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:25.597582    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:25.843855 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:25.614394    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 30 21:57:25.845098 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:25.471292    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:57:25.889986 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:25.889950    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:57:25.904643 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:25.904604    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:57:26.171161 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:26.167417    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 21:57:26.210285 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:26.210248    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:57:26.250369 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:26.234542    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:26.250369 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:26.237373    5087 scope.go:122] "RemoveContainer" containerID="daecc8c78236951ea4adf6d95ff7e0677cf5d6c30a103550aa9b914e9f20adb4"
Oct 30 21:57:26.702375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:26.689650    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc" podUID="f6a9f73b-bb1f-4270-a38d-3703c46fae07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:57:27.021372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:26.737140    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:57:27.021372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:26.774331    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml" podUID="01fdf382-6833-463a-bed9-c7c096166819" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Oct 30 21:57:27.021372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:26.873007    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Oct 30 21:57:27.021372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:26.873592    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 30 21:57:27.021372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:26.873652    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:27.021372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:26.873686    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:27.127468 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:26.913761    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:27.127751 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:26.914091    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:57:27.127892 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:26.961477    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:27.209713 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.197820    5087 scope.go:122] "RemoveContainer" containerID="700236ccc2d49227d83211fba18cb2b2ec3caf35823ede2b9fdb2255590fc603"
Oct 30 21:57:27.273435 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.262941    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:27.318813 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.262979    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/health\": context deadline exceeded"
Oct 30 21:57:27.319128 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.281503    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:27.319219 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.289329    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:57:27.319376 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.317553    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:57:27.326197 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.326166    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:57:27.329093 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.327613    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:57:27.329093 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.327983    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:57:27.381096 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.364038    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:57:27.381096 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.366103    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:57:27.381329 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.376082    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:57:27.386262 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.386245    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:57:27.386594 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.386512    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:57:27.386594 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.386564    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:57:27.386594 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.386597    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:57:27.386849 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.386794    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988e89a-g765q"
Oct 30 21:57:27.406394 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.391724    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:57:27.406394 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.392958    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:57:27.406394 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.393035    5087 scope.go:122] "RemoveContainer" containerID="00eb7a48a5a4d2b8fee65f8009228b538dd21739dde7866666a150fbeec2f00c"
Oct 30 21:57:27.407807 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.407611    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:57:27.420994 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.420864    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:57:27.421279 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.421256    5087 scope.go:122] "RemoveContainer" containerID="a5058454eca51f93d711cf1b7645ef3cab56c0e6b6a122c939612e43f12fb5cb"
Oct 30 21:57:27.433988 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.433971    5087 scope.go:122] "RemoveContainer" containerID="cd74c05cfcb78e15094aaac7f35f5b1b648f54f4d52ee1e6e2de882b86c23c85"
Oct 30 21:57:27.444221 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.444198    5087 scope.go:122] "RemoveContainer" containerID="50418632b93d4ba8db820f4ce87a9dce4db1577c82a41899a6aa09cd7f164caf"
Oct 30 21:57:27.453866 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.453845    5087 scope.go:122] "RemoveContainer" containerID="e416afe14a767908949fef63a5a11dc75ad74d8a6391e2b318ed51e26a10dd8b"
Oct 30 21:57:27.462885 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.462869    5087 scope.go:122] "RemoveContainer" containerID="f149155a7f8011fcc1b3c5eef4113f4d66a89abaeaf5fb1f359bc8fd3b8e0a8b"
Oct 30 21:57:27.473456 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.473440    5087 scope.go:122] "RemoveContainer" containerID="3502f71574093a6a118ae4200d7d5080456f779a29b1250e4c51a5101db2d6d4"
Oct 30 21:57:27.482892 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.482874    5087 scope.go:122] "RemoveContainer" containerID="f4ec1253e27a340249567ffa6d83b5e03e55068823837a1332dcd6007c6bc6c0"
Oct 30 21:57:27.493384 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.492454    5087 scope.go:122] "RemoveContainer" containerID="2cfcfe109e5d99f838646ede81ed18906c9f868dff6e2f020d9f7dcec651ddcf"
Oct 30 21:57:27.516191 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.516153    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8d32e3b-810b-40d4-a5bd-73b119077b30-kube-api-access-27kht\" (UniqueName: \"kubernetes.io/projected/b8d32e3b-810b-40d4-a5bd-73b119077b30-kube-api-access-27kht\") pod \"b8d32e3b-810b-40d4-a5bd-73b119077b30\" (UID: \"b8d32e3b-810b-40d4-a5bd-73b119077b30\") "
Oct 30 21:57:27.516435 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.516278    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8d32e3b-810b-40d4-a5bd-73b119077b30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d32e3b-810b-40d4-a5bd-73b119077b30-maas-api-openshift-ca\") pod \"b8d32e3b-810b-40d4-a5bd-73b119077b30\" (UID: \"b8d32e3b-810b-40d4-a5bd-73b119077b30\") "
Oct 30 21:57:27.516549 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.516524    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8d32e3b-810b-40d4-a5bd-73b119077b30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d32e3b-810b-40d4-a5bd-73b119077b30-maas-api-xks-ca\") pod \"b8d32e3b-810b-40d4-a5bd-73b119077b30\" (UID: \"b8d32e3b-810b-40d4-a5bd-73b119077b30\") "
Oct 30 21:57:27.519913 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.519499    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8d32e3b-810b-40d4-a5bd-73b119077b30-maas-api-openshift-ca" pod "b8d32e3b-810b-40d4-a5bd-73b119077b30" (UID: "b8d32e3b-810b-40d4-a5bd-73b119077b30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:27.525085 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.523488    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8d32e3b-810b-40d4-a5bd-73b119077b30-maas-api-xks-ca" pod "b8d32e3b-810b-40d4-a5bd-73b119077b30" (UID: "b8d32e3b-810b-40d4-a5bd-73b119077b30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:27.525613 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.525589    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8d32e3b-810b-40d4-a5bd-73b119077b30-kube-api-access-27kht" pod "b8d32e3b-810b-40d4-a5bd-73b119077b30" (UID: "b8d32e3b-810b-40d4-a5bd-73b119077b30"). InnerVolumeSpecName "kube-api-access-27kht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:27.625842 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.625798    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8d32e3b-810b-40d4-a5bd-73b119077b30-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:57:27.625842 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.625832    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27kht\" (UniqueName: \"kubernetes.io/projected/b8d32e3b-810b-40d4-a5bd-73b119077b30-kube-api-access-27kht\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:57:27.625842 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:27.625849    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8d32e3b-810b-40d4-a5bd-73b119077b30-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:57:28.131816 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:28.131782    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-744fc02d-zlt24"]
Oct 30 21:57:28.132960 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:28.132932    5087 cpu_manager.go:414] "containerMap: removing container" podUID="b8d32e3b-810b-40d4-a5bd-73b119077b30" containerName="revoke-keys"
Oct 30 21:57:28.132960 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:28.132961    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8d32e3b-810b-40d4-a5bd-73b119077b30" containerName="revoke-keys"
Oct 30 21:57:28.133173 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:28.133157    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8d32e3b-810b-40d4-a5bd-73b119077b30" containerName="revoke-keys"
Oct 30 21:57:28.140398 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:28.140367    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744fc02d-zlt24"
Oct 30 21:57:28.153062 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:28.153026    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2ct7\" (UniqueName: \"kubernetes.io/projected/687b2968-74a8-4969-9689-3050cf8415a7-kube-api-access-q2ct7\") pod \"maas-api-revoke-sub-744fc02d-zlt24\" (UID: \"687b2968-74a8-4969-9689-3050cf8415a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744fc02d-zlt24"
Oct 30 21:57:28.153207 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:28.153184    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/687b2968-74a8-4969-9689-3050cf8415a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-744fc02d-zlt24\" (UID: \"687b2968-74a8-4969-9689-3050cf8415a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744fc02d-zlt24"
Oct 30 21:57:28.153314 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:28.153294    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/687b2968-74a8-4969-9689-3050cf8415a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-744fc02d-zlt24\" (UID: \"687b2968-74a8-4969-9689-3050cf8415a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744fc02d-zlt24"
Oct 30 21:57:28.208747 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:28.208718    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-744fc02d-zlt24"]
Oct 30 21:57:28.255299 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:28.255255    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/687b2968-74a8-4969-9689-3050cf8415a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-744fc02d-zlt24\" (UID: \"687b2968-74a8-4969-9689-3050cf8415a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744fc02d-zlt24"
Oct 30 21:57:28.255545 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:28.255376    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/687b2968-74a8-4969-9689-3050cf8415a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-744fc02d-zlt24\" (UID: \"687b2968-74a8-4969-9689-3050cf8415a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744fc02d-zlt24"
Oct 30 21:57:28.255651 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:28.255634    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2ct7\" (UniqueName: \"kubernetes.io/projected/687b2968-74a8-4969-9689-3050cf8415a7-kube-api-access-q2ct7\") pod \"maas-api-revoke-sub-744fc02d-zlt24\" (UID: \"687b2968-74a8-4969-9689-3050cf8415a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744fc02d-zlt24"
Oct 30 21:57:28.256687 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:28.256653    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/687b2968-74a8-4969-9689-3050cf8415a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-744fc02d-zlt24\" (UID: \"687b2968-74a8-4969-9689-3050cf8415a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744fc02d-zlt24"
Oct 30 21:57:28.258719 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:28.258675    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/687b2968-74a8-4969-9689-3050cf8415a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-744fc02d-zlt24\" (UID: \"687b2968-74a8-4969-9689-3050cf8415a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744fc02d-zlt24"
Oct 30 21:57:28.334186 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:28.334142    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2ct7\" (UniqueName: \"kubernetes.io/projected/687b2968-74a8-4969-9689-3050cf8415a7-kube-api-access-q2ct7\") pod \"maas-api-revoke-sub-744fc02d-zlt24\" (UID: \"687b2968-74a8-4969-9689-3050cf8415a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-744fc02d-zlt24"
Oct 30 21:57:28.405588 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:28.405504    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988e89a-g765q"
Oct 30 21:57:28.410518 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:28.410488    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988e89a-g765q" event={"ID":"b8d32e3b-810b-40d4-a5bd-73b119077b30","Type":"ContainerDied","Data":"3cc2b7696bcb0bdbac430124ba2839eed086977ff8493d49cbe05d0148966b19"}
Oct 30 21:57:28.410702 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:28.410683    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cc2b7696bcb0bdbac430124ba2839eed086977ff8493d49cbe05d0148966b19"
Oct 30 21:57:28.460563 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:28.460529    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744fc02d-zlt24"
Oct 30 21:57:28.912453 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:28.912421    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f05b6443-fbbf556dc-bdtcf"]
Oct 30 21:57:28.919036 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:28.919011    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f05b6443-fbbf556dc-bdtcf"
Oct 30 21:57:29.037425 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:29.037010    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98f5afa5-3870-4279-b63f-d5971ab0c905-plugins-config-volume\") pod \"payload-processing-e2e-ait-f05b6443-fbbf556dc-bdtcf\" (UID: \"98f5afa5-3870-4279-b63f-d5971ab0c905\") " pod="openshift-ingress/payload-processing-e2e-ait-f05b6443-fbbf556dc-bdtcf"
Oct 30 21:57:29.037425 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:29.037048    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4n9z\" (UniqueName: \"kubernetes.io/projected/98f5afa5-3870-4279-b63f-d5971ab0c905-kube-api-access-n4n9z\") pod \"payload-processing-e2e-ait-f05b6443-fbbf556dc-bdtcf\" (UID: \"98f5afa5-3870-4279-b63f-d5971ab0c905\") " pod="openshift-ingress/payload-processing-e2e-ait-f05b6443-fbbf556dc-bdtcf"
Oct 30 21:57:29.046100 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:29.046069    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-f05b6443-dockercfg-phmkv\""
Oct 30 21:57:29.047173 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:29.046984    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-f05b6443\""
Oct 30 21:57:29.058802 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:29.058764    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f05b6443-fbbf556dc-bdtcf"]
Oct 30 21:57:29.138452 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:29.138413    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98f5afa5-3870-4279-b63f-d5971ab0c905-plugins-config-volume\") pod \"payload-processing-e2e-ait-f05b6443-fbbf556dc-bdtcf\" (UID: \"98f5afa5-3870-4279-b63f-d5971ab0c905\") " pod="openshift-ingress/payload-processing-e2e-ait-f05b6443-fbbf556dc-bdtcf"
Oct 30 21:57:29.138452 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:29.138448    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4n9z\" (UniqueName: \"kubernetes.io/projected/98f5afa5-3870-4279-b63f-d5971ab0c905-kube-api-access-n4n9z\") pod \"payload-processing-e2e-ait-f05b6443-fbbf556dc-bdtcf\" (UID: \"98f5afa5-3870-4279-b63f-d5971ab0c905\") " pod="openshift-ingress/payload-processing-e2e-ait-f05b6443-fbbf556dc-bdtcf"
Oct 30 21:57:29.139202 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:29.139177    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98f5afa5-3870-4279-b63f-d5971ab0c905-plugins-config-volume\") pod \"payload-processing-e2e-ait-f05b6443-fbbf556dc-bdtcf\" (UID: \"98f5afa5-3870-4279-b63f-d5971ab0c905\") " pod="openshift-ingress/payload-processing-e2e-ait-f05b6443-fbbf556dc-bdtcf"
Oct 30 21:57:29.156899 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:29.156866    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-744fc02d-zlt24"]
Oct 30 21:57:29.157364 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:57:29.157323    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod687b2968_74a8_4969_9689_3050cf8415a7.slice/crio-2b911d6a7beb76fe71fdca7c5491379e159d2c89fd6b21924a9e4cb725ec6b7f WatchSource:0}: Error finding container 2b911d6a7beb76fe71fdca7c5491379e159d2c89fd6b21924a9e4cb725ec6b7f: Status 404 returned error can't find the container with id 2b911d6a7beb76fe71fdca7c5491379e159d2c89fd6b21924a9e4cb725ec6b7f
Oct 30 21:57:29.196221 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:29.195836    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4n9z\" (UniqueName: \"kubernetes.io/projected/98f5afa5-3870-4279-b63f-d5971ab0c905-kube-api-access-n4n9z\") pod \"payload-processing-e2e-ait-f05b6443-fbbf556dc-bdtcf\" (UID: \"98f5afa5-3870-4279-b63f-d5971ab0c905\") " pod="openshift-ingress/payload-processing-e2e-ait-f05b6443-fbbf556dc-bdtcf"
Oct 30 21:57:29.243392 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:29.243326    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f05b6443-fbbf556dc-bdtcf"
Oct 30 21:57:29.421928 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:29.421895    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744fc02d-zlt24" event={"ID":"687b2968-74a8-4969-9689-3050cf8415a7","Type":"ContainerStarted","Data":"2b911d6a7beb76fe71fdca7c5491379e159d2c89fd6b21924a9e4cb725ec6b7f"}
Oct 30 21:57:29.657140 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:29.650397    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f05b6443-fbbf556dc-bdtcf"]
Oct 30 21:57:30.431585 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:30.431544    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f05b6443-fbbf556dc-bdtcf" event={"ID":"98f5afa5-3870-4279-b63f-d5971ab0c905","Type":"ContainerStarted","Data":"3631a19f9c0aaac24db8f4581fddd7f843e28f79d9928ff90e1d2132a679db8e"}
Oct 30 21:57:30.432084 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:30.431614    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f05b6443-fbbf556dc-bdtcf" event={"ID":"98f5afa5-3870-4279-b63f-d5971ab0c905","Type":"ContainerStarted","Data":"b924d8f274cd14fd3bd091c3f43f5de74bdc21367a40cf4e2434af72152c56fc"}
Oct 30 21:57:30.432084 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:30.431645    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-f05b6443-fbbf556dc-bdtcf"
Oct 30 21:57:30.433675 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:30.433649    5087 generic.go:356] "Generic (PLEG): container finished" podID="687b2968-74a8-4969-9689-3050cf8415a7" containerID="049e438fadb172173726c66e97e5da3bb67e3c3c60108ca22738eac1318c5509" exitCode=0
Oct 30 21:57:30.433808 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:30.433701    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744fc02d-zlt24" event={"ID":"687b2968-74a8-4969-9689-3050cf8415a7","Type":"ContainerDied","Data":"049e438fadb172173726c66e97e5da3bb67e3c3c60108ca22738eac1318c5509"}
Oct 30 21:57:30.546089 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:30.546024    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-f05b6443-fbbf556dc-bdtcf" podStartSLOduration=2.546005789 podStartE2EDuration="2.546005789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:30.545532418 +0000 UTC m=+1445.020036399" watchObservedRunningTime="2026-10-30 21:57:30.546005789 +0000 UTC m=+1445.020509768"
Oct 30 21:57:31.640518 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:31.640487    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744fc02d-zlt24"
Oct 30 21:57:31.786709 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:31.786680    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/687b2968-74a8-4969-9689-3050cf8415a7-kube-api-access-q2ct7\" (UniqueName: \"kubernetes.io/projected/687b2968-74a8-4969-9689-3050cf8415a7-kube-api-access-q2ct7\") pod \"687b2968-74a8-4969-9689-3050cf8415a7\" (UID: \"687b2968-74a8-4969-9689-3050cf8415a7\") "
Oct 30 21:57:31.786902 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:31.786784    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/687b2968-74a8-4969-9689-3050cf8415a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/687b2968-74a8-4969-9689-3050cf8415a7-maas-api-xks-ca\") pod \"687b2968-74a8-4969-9689-3050cf8415a7\" (UID: \"687b2968-74a8-4969-9689-3050cf8415a7\") "
Oct 30 21:57:31.786980 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:31.786955    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/687b2968-74a8-4969-9689-3050cf8415a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/687b2968-74a8-4969-9689-3050cf8415a7-maas-api-openshift-ca\") pod \"687b2968-74a8-4969-9689-3050cf8415a7\" (UID: \"687b2968-74a8-4969-9689-3050cf8415a7\") "
Oct 30 21:57:31.787401 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:31.787373    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/687b2968-74a8-4969-9689-3050cf8415a7-maas-api-openshift-ca" pod "687b2968-74a8-4969-9689-3050cf8415a7" (UID: "687b2968-74a8-4969-9689-3050cf8415a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:31.787698 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:31.787673    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/687b2968-74a8-4969-9689-3050cf8415a7-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:57:31.789257 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:31.789234    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/687b2968-74a8-4969-9689-3050cf8415a7-maas-api-xks-ca" pod "687b2968-74a8-4969-9689-3050cf8415a7" (UID: "687b2968-74a8-4969-9689-3050cf8415a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:31.789565 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:31.789543    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/687b2968-74a8-4969-9689-3050cf8415a7-kube-api-access-q2ct7" pod "687b2968-74a8-4969-9689-3050cf8415a7" (UID: "687b2968-74a8-4969-9689-3050cf8415a7"). InnerVolumeSpecName "kube-api-access-q2ct7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:31.888650 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:31.888609    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/687b2968-74a8-4969-9689-3050cf8415a7-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:57:31.888650 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:31.888643    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2ct7\" (UniqueName: \"kubernetes.io/projected/687b2968-74a8-4969-9689-3050cf8415a7-kube-api-access-q2ct7\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:57:31.897580 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:31.897545    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc252a9a-8g9l7"]
Oct 30 21:57:31.898279 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:31.898261    5087 cpu_manager.go:414] "containerMap: removing container" podUID="687b2968-74a8-4969-9689-3050cf8415a7" containerName="revoke-keys"
Oct 30 21:57:31.898327 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:31.898284    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="687b2968-74a8-4969-9689-3050cf8415a7" containerName="revoke-keys"
Oct 30 21:57:31.898499 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:31.898485    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="687b2968-74a8-4969-9689-3050cf8415a7" containerName="revoke-keys"
Oct 30 21:57:31.902794 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:31.902773    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc252a9a-8g9l7"
Oct 30 21:57:31.960215 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:31.960183    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc252a9a-8g9l7"]
Oct 30 21:57:32.091774 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:32.091683    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b52a697c-969c-4df2-84ce-80e2a399ffcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc252a9a-8g9l7\" (UID: \"b52a697c-969c-4df2-84ce-80e2a399ffcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc252a9a-8g9l7"
Oct 30 21:57:32.091774 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:32.091750    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5psdl\" (UniqueName: \"kubernetes.io/projected/b52a697c-969c-4df2-84ce-80e2a399ffcb-kube-api-access-5psdl\") pod \"maas-api-revoke-sub-bc252a9a-8g9l7\" (UID: \"b52a697c-969c-4df2-84ce-80e2a399ffcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc252a9a-8g9l7"
Oct 30 21:57:32.092023 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:32.091892    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b52a697c-969c-4df2-84ce-80e2a399ffcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc252a9a-8g9l7\" (UID: \"b52a697c-969c-4df2-84ce-80e2a399ffcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc252a9a-8g9l7"
Oct 30 21:57:32.193387 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:32.193300    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b52a697c-969c-4df2-84ce-80e2a399ffcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc252a9a-8g9l7\" (UID: \"b52a697c-969c-4df2-84ce-80e2a399ffcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc252a9a-8g9l7"
Oct 30 21:57:32.193562 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:32.193441    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5psdl\" (UniqueName: \"kubernetes.io/projected/b52a697c-969c-4df2-84ce-80e2a399ffcb-kube-api-access-5psdl\") pod \"maas-api-revoke-sub-bc252a9a-8g9l7\" (UID: \"b52a697c-969c-4df2-84ce-80e2a399ffcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc252a9a-8g9l7"
Oct 30 21:57:32.193562 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:32.193506    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b52a697c-969c-4df2-84ce-80e2a399ffcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc252a9a-8g9l7\" (UID: \"b52a697c-969c-4df2-84ce-80e2a399ffcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc252a9a-8g9l7"
Oct 30 21:57:32.194236 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:32.194211    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b52a697c-969c-4df2-84ce-80e2a399ffcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc252a9a-8g9l7\" (UID: \"b52a697c-969c-4df2-84ce-80e2a399ffcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc252a9a-8g9l7"
Oct 30 21:57:32.195992 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:32.195968    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b52a697c-969c-4df2-84ce-80e2a399ffcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc252a9a-8g9l7\" (UID: \"b52a697c-969c-4df2-84ce-80e2a399ffcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc252a9a-8g9l7"
Oct 30 21:57:32.251983 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:32.251955    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5psdl\" (UniqueName: \"kubernetes.io/projected/b52a697c-969c-4df2-84ce-80e2a399ffcb-kube-api-access-5psdl\") pod \"maas-api-revoke-sub-bc252a9a-8g9l7\" (UID: \"b52a697c-969c-4df2-84ce-80e2a399ffcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc252a9a-8g9l7"
Oct 30 21:57:32.452552 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:32.452470    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744fc02d-zlt24"
Oct 30 21:57:32.452833 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:32.452494    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-744fc02d-zlt24" event={"ID":"687b2968-74a8-4969-9689-3050cf8415a7","Type":"ContainerDied","Data":"2b911d6a7beb76fe71fdca7c5491379e159d2c89fd6b21924a9e4cb725ec6b7f"}
Oct 30 21:57:32.452985 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:32.452861    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b911d6a7beb76fe71fdca7c5491379e159d2c89fd6b21924a9e4cb725ec6b7f"
Oct 30 21:57:32.520095 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:32.520062    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc252a9a-8g9l7"
Oct 30 21:57:32.904547 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:32.904519    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc252a9a-8g9l7"]
Oct 30 21:57:32.905133 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:57:32.905109    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb52a697c_969c_4df2_84ce_80e2a399ffcb.slice/crio-cff497278bba109f16f9d89865bffc7cd9e91e0ebbd524c64061439a8f855b91 WatchSource:0}: Error finding container cff497278bba109f16f9d89865bffc7cd9e91e0ebbd524c64061439a8f855b91: Status 404 returned error can't find the container with id cff497278bba109f16f9d89865bffc7cd9e91e0ebbd524c64061439a8f855b91
Oct 30 21:57:33.469200 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:33.469161    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc252a9a-8g9l7" event={"ID":"b52a697c-969c-4df2-84ce-80e2a399ffcb","Type":"ContainerStarted","Data":"a52679231d012bc4dbb194c078a18b81ed99b5de68c9994f845b379bc01caf62"}
Oct 30 21:57:33.469516 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:33.469226    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc252a9a-8g9l7" event={"ID":"b52a697c-969c-4df2-84ce-80e2a399ffcb","Type":"ContainerStarted","Data":"cff497278bba109f16f9d89865bffc7cd9e91e0ebbd524c64061439a8f855b91"}
Oct 30 21:57:33.589039 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:33.588987    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc252a9a-8g9l7" podStartSLOduration=2.5889743469999997 podStartE2EDuration="2.588974347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:33.58380014 +0000 UTC m=+1448.058304118" watchObservedRunningTime="2026-10-30 21:57:33.588974347 +0000 UTC m=+1448.063478352"
Oct 30 21:57:34.481452 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:34.481419    5087 generic.go:356] "Generic (PLEG): container finished" podID="b52a697c-969c-4df2-84ce-80e2a399ffcb" containerID="a52679231d012bc4dbb194c078a18b81ed99b5de68c9994f845b379bc01caf62" exitCode=0
Oct 30 21:57:34.481911 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:34.481507    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc252a9a-8g9l7" event={"ID":"b52a697c-969c-4df2-84ce-80e2a399ffcb","Type":"ContainerDied","Data":"a52679231d012bc4dbb194c078a18b81ed99b5de68c9994f845b379bc01caf62"}
Oct 30 21:57:35.665619 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:35.665592    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc252a9a-8g9l7"
Oct 30 21:57:35.751803 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:35.751722    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b52a697c-969c-4df2-84ce-80e2a399ffcb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b52a697c-969c-4df2-84ce-80e2a399ffcb-maas-api-xks-ca\") pod \"b52a697c-969c-4df2-84ce-80e2a399ffcb\" (UID: \"b52a697c-969c-4df2-84ce-80e2a399ffcb\") "
Oct 30 21:57:35.751974 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:35.751856    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b52a697c-969c-4df2-84ce-80e2a399ffcb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b52a697c-969c-4df2-84ce-80e2a399ffcb-maas-api-openshift-ca\") pod \"b52a697c-969c-4df2-84ce-80e2a399ffcb\" (UID: \"b52a697c-969c-4df2-84ce-80e2a399ffcb\") "
Oct 30 21:57:35.751974 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:35.751926    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b52a697c-969c-4df2-84ce-80e2a399ffcb-kube-api-access-5psdl\" (UniqueName: \"kubernetes.io/projected/b52a697c-969c-4df2-84ce-80e2a399ffcb-kube-api-access-5psdl\") pod \"b52a697c-969c-4df2-84ce-80e2a399ffcb\" (UID: \"b52a697c-969c-4df2-84ce-80e2a399ffcb\") "
Oct 30 21:57:35.752221 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:35.752179    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b52a697c-969c-4df2-84ce-80e2a399ffcb-maas-api-openshift-ca" pod "b52a697c-969c-4df2-84ce-80e2a399ffcb" (UID: "b52a697c-969c-4df2-84ce-80e2a399ffcb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:35.752554 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:35.752531    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b52a697c-969c-4df2-84ce-80e2a399ffcb-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:57:35.754084 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:35.754062    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b52a697c-969c-4df2-84ce-80e2a399ffcb-maas-api-xks-ca" pod "b52a697c-969c-4df2-84ce-80e2a399ffcb" (UID: "b52a697c-969c-4df2-84ce-80e2a399ffcb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:35.754451 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:35.754426    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b52a697c-969c-4df2-84ce-80e2a399ffcb-kube-api-access-5psdl" pod "b52a697c-969c-4df2-84ce-80e2a399ffcb" (UID: "b52a697c-969c-4df2-84ce-80e2a399ffcb"). InnerVolumeSpecName "kube-api-access-5psdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:35.853733 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:35.853684    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5psdl\" (UniqueName: \"kubernetes.io/projected/b52a697c-969c-4df2-84ce-80e2a399ffcb-kube-api-access-5psdl\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:57:35.853733 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:35.853732    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b52a697c-969c-4df2-84ce-80e2a399ffcb-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:57:36.522067 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:36.522030    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc252a9a-8g9l7" event={"ID":"b52a697c-969c-4df2-84ce-80e2a399ffcb","Type":"ContainerDied","Data":"cff497278bba109f16f9d89865bffc7cd9e91e0ebbd524c64061439a8f855b91"}
Oct 30 21:57:36.522249 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:36.522109    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cff497278bba109f16f9d89865bffc7cd9e91e0ebbd524c64061439a8f855b91"
Oct 30 21:57:36.522249 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:36.522181    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc252a9a-8g9l7"
Oct 30 21:57:36.840164 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:36.840073    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-be51942b-b8f988b87-bnklt"]
Oct 30 21:57:36.841007 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:36.840981    5087 cpu_manager.go:414] "containerMap: removing container" podUID="b52a697c-969c-4df2-84ce-80e2a399ffcb" containerName="revoke-keys"
Oct 30 21:57:36.841135 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:36.841009    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b52a697c-969c-4df2-84ce-80e2a399ffcb" containerName="revoke-keys"
Oct 30 21:57:36.841226 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:36.841201    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="b52a697c-969c-4df2-84ce-80e2a399ffcb" containerName="revoke-keys"
Oct 30 21:57:36.848988 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:36.847675    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-be51942b-b8f988b87-bnklt"
Oct 30 21:57:36.890739 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:36.890584    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f157ab83-6db1-4d6a-b9f5-57ef3b0aa752-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-be51942b-b8f988b87-bnklt\" (UID: \"f157ab83-6db1-4d6a-b9f5-57ef3b0aa752\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-be51942b-b8f988b87-bnklt"
Oct 30 21:57:36.890739 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:36.890684    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvqn9\" (UniqueName: \"kubernetes.io/projected/f157ab83-6db1-4d6a-b9f5-57ef3b0aa752-kube-api-access-zvqn9\") pod \"payload-pre-processing-e2e-ait-be51942b-b8f988b87-bnklt\" (UID: \"f157ab83-6db1-4d6a-b9f5-57ef3b0aa752\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-be51942b-b8f988b87-bnklt"
Oct 30 21:57:36.905372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:36.902582    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-be51942b\""
Oct 30 21:57:36.905372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:36.902778    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-be51942b-dockercfg-224x9\""
Oct 30 21:57:36.927914 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:36.927871    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-be51942b-b8f988b87-bnklt"]
Oct 30 21:57:37.005434 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:36.995318    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f157ab83-6db1-4d6a-b9f5-57ef3b0aa752-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-be51942b-b8f988b87-bnklt\" (UID: \"f157ab83-6db1-4d6a-b9f5-57ef3b0aa752\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-be51942b-b8f988b87-bnklt"
Oct 30 21:57:37.005434 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:36.995479    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvqn9\" (UniqueName: \"kubernetes.io/projected/f157ab83-6db1-4d6a-b9f5-57ef3b0aa752-kube-api-access-zvqn9\") pod \"payload-pre-processing-e2e-ait-be51942b-b8f988b87-bnklt\" (UID: \"f157ab83-6db1-4d6a-b9f5-57ef3b0aa752\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-be51942b-b8f988b87-bnklt"
Oct 30 21:57:37.005434 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:37.004447    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f157ab83-6db1-4d6a-b9f5-57ef3b0aa752-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-be51942b-b8f988b87-bnklt\" (UID: \"f157ab83-6db1-4d6a-b9f5-57ef3b0aa752\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-be51942b-b8f988b87-bnklt"
Oct 30 21:57:37.060612 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:37.060578    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvqn9\" (UniqueName: \"kubernetes.io/projected/f157ab83-6db1-4d6a-b9f5-57ef3b0aa752-kube-api-access-zvqn9\") pod \"payload-pre-processing-e2e-ait-be51942b-b8f988b87-bnklt\" (UID: \"f157ab83-6db1-4d6a-b9f5-57ef3b0aa752\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-be51942b-b8f988b87-bnklt"
Oct 30 21:57:37.226905 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:37.226813    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-be51942b-b8f988b87-bnklt"
Oct 30 21:57:37.698728 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:37.698534    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-be51942b-b8f988b87-bnklt"]
Oct 30 21:57:37.702833 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:57:37.702802    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf157ab83_6db1_4d6a_b9f5_57ef3b0aa752.slice/crio-62a99e8fe0b3a41391b6906f83c3580846e8799730d6f607d75214642eb4ce2e WatchSource:0}: Error finding container 62a99e8fe0b3a41391b6906f83c3580846e8799730d6f607d75214642eb4ce2e: Status 404 returned error can't find the container with id 62a99e8fe0b3a41391b6906f83c3580846e8799730d6f607d75214642eb4ce2e
Oct 30 21:57:38.659914 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:38.659870    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-be51942b-b8f988b87-bnklt" event={"ID":"f157ab83-6db1-4d6a-b9f5-57ef3b0aa752","Type":"ContainerStarted","Data":"45d961f2b7d1ab8044f09b721031fec5c97dca70f2327897eebc6704f43d89b8"}
Oct 30 21:57:38.660415 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:38.659943    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-be51942b-b8f988b87-bnklt" event={"ID":"f157ab83-6db1-4d6a-b9f5-57ef3b0aa752","Type":"ContainerStarted","Data":"62a99e8fe0b3a41391b6906f83c3580846e8799730d6f607d75214642eb4ce2e"}
Oct 30 21:57:38.660415 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:38.659976    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-be51942b-b8f988b87-bnklt"
Oct 30 21:57:38.783274 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:38.783213    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-be51942b-b8f988b87-bnklt" podStartSLOduration=2.783194999 podStartE2EDuration="2.783194999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:38.775513161 +0000 UTC m=+1453.250017141" watchObservedRunningTime="2026-10-30 21:57:38.783194999 +0000 UTC m=+1453.257698985"
Oct 30 21:57:41.445285 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:41.445257    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-f05b6443-fbbf556dc-bdtcf"
Oct 30 21:57:45.456390 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.456309    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"]
Oct 30 21:57:45.460765 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.460741    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:45.502008 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.501978    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-workload-socket\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:45.502130 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.502013    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/894d8554-651d-4ad9-8521-da6c98edca6d-istiod-ca-cert\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:45.502166 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.502138    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-workload-certs\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:45.502213 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.502197    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/894d8554-651d-4ad9-8521-da6c98edca6d-istio-token\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:45.502252 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.502240    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-credential-socket\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:45.502290 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.502269    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-istio-data\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:45.502388 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.502373    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n4xw\" (UniqueName: \"kubernetes.io/projected/894d8554-651d-4ad9-8521-da6c98edca6d-kube-api-access-2n4xw\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:45.502467 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.502446    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/894d8554-651d-4ad9-8521-da6c98edca6d-istio-podinfo\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:45.502522 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.502506    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-istio-envoy\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:45.508193 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.508162    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"]
Oct 30 21:57:45.513704 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.513684    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:45.557947 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.557863    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"]
Oct 30 21:57:45.570031 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.569994    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"]
Oct 30 21:57:45.603479 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.603445    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n4xw\" (UniqueName: \"kubernetes.io/projected/894d8554-651d-4ad9-8521-da6c98edca6d-kube-api-access-2n4xw\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:45.603675 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.603524    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/894d8554-651d-4ad9-8521-da6c98edca6d-istio-podinfo\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:45.603675 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.603556    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-istio-envoy\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:45.603675 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.603618    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89d4443a-0fb1-461e-a866-05351817130e-istio-podinfo\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:45.603675 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.603656    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89d4443a-0fb1-461e-a866-05351817130e-istio-token\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:45.603907 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.603690    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-istio-envoy\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:45.603907 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.603735    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-workload-socket\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:45.603907 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.603763    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/894d8554-651d-4ad9-8521-da6c98edca6d-istiod-ca-cert\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:45.604165 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.604138    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-workload-socket\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:45.604719 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.604692    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/894d8554-651d-4ad9-8521-da6c98edca6d-istiod-ca-cert\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:45.604894 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.604868    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-workload-certs\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:45.605013 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.604951    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffn4x\" (UniqueName: \"kubernetes.io/projected/89d4443a-0fb1-461e-a866-05351817130e-kube-api-access-ffn4x\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:45.605013 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.605008    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-credential-socket\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:45.605147 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.605071    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-istio-data\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:45.605213 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.605155    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-workload-socket\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:45.605213 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.605196    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-workload-certs\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:45.605317 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.605292    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/894d8554-651d-4ad9-8521-da6c98edca6d-istio-token\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:45.605396 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.605323    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-credential-socket\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:45.605450 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.605412    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-istio-data\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:45.605561 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.605465    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89d4443a-0fb1-461e-a866-05351817130e-istiod-ca-cert\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:45.605899 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.605876    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-workload-certs\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:45.606468 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.606387    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-credential-socket\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:45.606687 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.606661    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-istio-data\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:45.607454 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.607430    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-istio-envoy\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:45.607901 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.607877    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/894d8554-651d-4ad9-8521-da6c98edca6d-istio-podinfo\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:45.665915 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.665878    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n4xw\" (UniqueName: \"kubernetes.io/projected/894d8554-651d-4ad9-8521-da6c98edca6d-kube-api-access-2n4xw\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:45.666997 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.666973    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/894d8554-651d-4ad9-8521-da6c98edca6d-istio-token\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:45.706625 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.706554    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-workload-socket\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:45.706751 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.706682    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89d4443a-0fb1-461e-a866-05351817130e-istiod-ca-cert\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:45.706817 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.706801    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89d4443a-0fb1-461e-a866-05351817130e-istio-podinfo\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:45.706874 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.706838    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89d4443a-0fb1-461e-a866-05351817130e-istio-token\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:45.706915 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.706869    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-istio-envoy\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:45.706951 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.706926    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-workload-certs\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:45.706984 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.706942    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-workload-socket\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:45.706984 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.706954    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffn4x\" (UniqueName: \"kubernetes.io/projected/89d4443a-0fb1-461e-a866-05351817130e-kube-api-access-ffn4x\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:45.707046 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.707015    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-credential-socket\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:45.707076 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.707046    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-istio-data\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:45.707481 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.707455    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-credential-socket\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:45.707655 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.707479    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-workload-certs\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:45.707742 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.707502    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-istio-data\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:45.707742 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.707497    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89d4443a-0fb1-461e-a866-05351817130e-istiod-ca-cert\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:45.709514 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.709492    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-istio-envoy\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:45.709626 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.709581    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89d4443a-0fb1-461e-a866-05351817130e-istio-podinfo\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:45.767523 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.767497    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89d4443a-0fb1-461e-a866-05351817130e-istio-token\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:45.767670 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.767604    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffn4x\" (UniqueName: \"kubernetes.io/projected/89d4443a-0fb1-461e-a866-05351817130e-kube-api-access-ffn4x\") pod \"e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") " pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:45.788757 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.788735    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:45.829454 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:45.829425    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:46.238406 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:57:46.238374    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod894d8554_651d_4ad9_8521_da6c98edca6d.slice/crio-45a37f156b8f8e3b554bb6e2436998440de00e4db128b13d0dd65b40dedce169 WatchSource:0}: Error finding container 45a37f156b8f8e3b554bb6e2436998440de00e4db128b13d0dd65b40dedce169: Status 404 returned error can't find the container with id 45a37f156b8f8e3b554bb6e2436998440de00e4db128b13d0dd65b40dedce169
Oct 30 21:57:46.239979 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:46.239950    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"]
Oct 30 21:57:46.240980 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:46.240950    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:57:46.241055 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:46.241013    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:57:46.241055 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:46.241041    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:57:46.324878 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:46.324795    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"]
Oct 30 21:57:46.326338 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:57:46.326312    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d4443a_0fb1_461e_a866_05351817130e.slice/crio-78942715bc4cbc7489bf644a09e1a74f99ccc5cc4d58cbd4f841fd0ec61790bc WatchSource:0}: Error finding container 78942715bc4cbc7489bf644a09e1a74f99ccc5cc4d58cbd4f841fd0ec61790bc: Status 404 returned error can't find the container with id 78942715bc4cbc7489bf644a09e1a74f99ccc5cc4d58cbd4f841fd0ec61790bc
Oct 30 21:57:46.328940 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:46.328906    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:57:46.329026 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:46.328987    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:57:46.329079 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:46.329026    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:57:46.798566 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:46.798522    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg" event={"ID":"894d8554-651d-4ad9-8521-da6c98edca6d","Type":"ContainerStarted","Data":"44faaa6dc01eadd1454099f86f68e37eb2d12eae44ef8d995c62974668560deb"}
Oct 30 21:57:46.799043 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:46.798587    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg" event={"ID":"894d8554-651d-4ad9-8521-da6c98edca6d","Type":"ContainerStarted","Data":"45a37f156b8f8e3b554bb6e2436998440de00e4db128b13d0dd65b40dedce169"}
Oct 30 21:57:46.801856 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:46.801826    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k" event={"ID":"89d4443a-0fb1-461e-a866-05351817130e","Type":"ContainerStarted","Data":"a995abf6d93c7e70bb189419720531fb8bf63b9983121430844125d99718b834"}
Oct 30 21:57:46.802001 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:46.801880    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k" event={"ID":"89d4443a-0fb1-461e-a866-05351817130e","Type":"ContainerStarted","Data":"78942715bc4cbc7489bf644a09e1a74f99ccc5cc4d58cbd4f841fd0ec61790bc"}
Oct 30 21:57:46.829619 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:46.829574    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:46.960000 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:46.959945    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg" podStartSLOduration=1.95992954 podStartE2EDuration="1.95992954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:46.957701909 +0000 UTC m=+1461.432205900" watchObservedRunningTime="2026-10-30 21:57:46.95992954 +0000 UTC m=+1461.434433518"
Oct 30 21:57:47.496374 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:57:47.495535    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:57:47.804426 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:47.797634    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 21:57:47.804426 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:47.797692    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg" podUID="894d8554-651d-4ad9-8521-da6c98edca6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 21:57:47.804426 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:47.803109    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:47.847371 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:47.841754    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 21:57:47.847371 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:47.841812    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k" podUID="89d4443a-0fb1-461e-a866-05351817130e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 21:57:48.224435 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:48.223479    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": context deadline exceeded"
Oct 30 21:57:48.374571 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:48.374504    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:48.392495 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:48.392449    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:57:48.393116 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:48.393085    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:57:48.393626 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:48.393601    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:57:48.394023 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:48.394000    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:57:48.396092 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:48.394652    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:48.418375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:48.414460    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 21:57:48.454990 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:48.451846    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc" podUID="f6a9f73b-bb1f-4270-a38d-3703c46fae07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:48.477534 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:48.477434    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:48.478662 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:48.478610    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml" podUID="01fdf382-6833-463a-bed9-c7c096166819" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": context deadline exceeded"
Oct 30 21:57:48.795373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:48.792649    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 21:57:48.795373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:48.792720    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg" podUID="894d8554-651d-4ad9-8521-da6c98edca6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 21:57:48.799371 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:48.796415    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": context deadline exceeded"
Oct 30 21:57:48.831374 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:48.831026    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 21:57:48.831374 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:48.831092    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k" podUID="89d4443a-0fb1-461e-a866-05351817130e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 21:57:49.046659 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.046535    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:49.049569 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.046912    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:49.089376 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.046977    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:49.100426 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.067490    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:49.235372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.234451    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:49.235372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.235022    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:57:49.383613 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.383516    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:57:49.384217 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.384192    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:57:49.393435 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.393390    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:57:49.393961 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.393918    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 21:57:49.394434 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.394411    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:57:49.394981 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.394964    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:57:49.396662 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.395210    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:57:49.396871 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.396837    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:57:49.397051 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.397011    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 30 21:57:49.399770 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.399736    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:57:49.407110 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.407079    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:49.407487 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.407451    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:57:49.420528 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.420480    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:49.421078 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.421056    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:57:49.441483 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.435589    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k" podStartSLOduration=4.435567275 podStartE2EDuration="4.435567275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:47.121147793 +0000 UTC m=+1461.595651806" watchObservedRunningTime="2026-10-30 21:57:49.435567275 +0000 UTC m=+1463.910071256"
Oct 30 21:57:49.478473 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.478423    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:49.490459 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.479043    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml" podUID="01fdf382-6833-463a-bed9-c7c096166819" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:49.547375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.479124    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:57:49.547375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.479597    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc" podUID="f6a9f73b-bb1f-4270-a38d-3703c46fae07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:49.547375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.529279    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:57:49.547375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.529931    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:57:49.683816 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.649804    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:57:49.886424 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.795441    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:49.886424 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.796219    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded"
Oct 30 21:57:49.886424 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.797304    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 21:57:49.886424 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.797365    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg" podUID="894d8554-651d-4ad9-8521-da6c98edca6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 21:57:49.886424 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.832590    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 21:57:49.886424 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:49.832652    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k" podUID="89d4443a-0fb1-461e-a866-05351817130e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 21:57:50.116696 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:50.107790    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:50.116696 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:50.109574    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:50.235500 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:50.235458    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:50.420426 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:50.394767    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:50.420426 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:50.408268    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:57:50.420426 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:50.410532    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:50.420426 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:50.411819    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded"
Oct 30 21:57:50.420426 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:50.413552    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:50.444566 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:50.422087    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:50.517372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:50.515127    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:57:50.564339 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:50.564296    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc" podUID="f6a9f73b-bb1f-4270-a38d-3703c46fae07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": context deadline exceeded"
Oct 30 21:57:50.566972 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:50.564654    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml" podUID="01fdf382-6833-463a-bed9-c7c096166819" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:50.681521 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:50.681425    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 21:57:50.823418 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:50.812915    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 21:57:50.823418 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:50.812972    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg" podUID="894d8554-651d-4ad9-8521-da6c98edca6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 21:57:50.881371 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:50.875134    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 21:57:50.881371 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:50.875212    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k" podUID="89d4443a-0fb1-461e-a866-05351817130e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 21:57:50.927375 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:57:50.919785    5087 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.2s"
Oct 30 21:57:50.927375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:50.919907    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-be51942b-b8f988b87-bnklt"
Oct 30 21:57:50.927375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:50.919975    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:57:50.927375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:50.920015    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:57:51.125494 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:51.125445    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 30 21:57:51.159379 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:51.131750    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Oct 30 21:57:51.166580 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:51.165797    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 21:57:51.190003 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:51.189964    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:51.238447 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:51.238402    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:51.423429 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:51.415399    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:51.537621 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:51.537580    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 21:57:51.538081 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:51.424912    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:57:51.538519 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:51.424971    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 21:57:51.561413 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:51.425002    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:51.599594 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:51.599563    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 21:57:51.629707 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:51.615329    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 21:57:51.658373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:51.656619    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 21:57:51.681428 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:51.672374    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 21:57:51.681428 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:51.673269    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 21:57:51.695484 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:51.695444    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc" podUID="f6a9f73b-bb1f-4270-a38d-3703c46fae07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:51.701830 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:51.701471    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:57:51.707449 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:51.703881    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:57:51.707449 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:51.423686    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": context deadline exceeded"
Oct 30 21:57:51.717915 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:51.716789    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 21:57:51.798748 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:51.798709    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 21:57:51.798948 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:51.798770    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg" podUID="894d8554-651d-4ad9-8521-da6c98edca6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 21:57:51.802839 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:51.802813    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 21:57:51.802990 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:51.802864    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 21:57:51.802990 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:51.802895    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 21:57:51.805934 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:51.805913    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 21:57:51.806404 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:51.806383    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 21:57:51.806738 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:51.806719    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 21:57:51.835521 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:51.835490    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 21:57:51.845515 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:51.836416    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k" podUID="89d4443a-0fb1-461e-a866-05351817130e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 21:57:52.792475 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:52.789124    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 21:57:52.792475 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:52.789182    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg" podUID="894d8554-651d-4ad9-8521-da6c98edca6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 21:57:52.830597 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:52.830566    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 21:57:52.830869 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:52.830840    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k" podUID="89d4443a-0fb1-461e-a866-05351817130e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 21:57:53.802390 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:53.795189    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 21:57:53.803176 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:53.803144    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg" podUID="894d8554-651d-4ad9-8521-da6c98edca6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 21:57:53.829825 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:53.829793    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 21:57:53.829990 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:53.829849    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k" podUID="89d4443a-0fb1-461e-a866-05351817130e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 21:57:54.789451 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:54.789049    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 21:57:54.789451 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:54.789107    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg" podUID="894d8554-651d-4ad9-8521-da6c98edca6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 21:57:54.830724 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:54.829772    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 21:57:54.830724 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:54.829846    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k" podUID="89d4443a-0fb1-461e-a866-05351817130e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 21:57:54.836174 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:54.836147    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7057-292d8a"]
Oct 30 21:57:54.885375 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:54.884722    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7057-292d8a"
Oct 30 21:57:54.896522 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:54.896496    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7057-292d8a"]
Oct 30 21:57:54.927740 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:54.927686    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqjp5\" (UniqueName: \"kubernetes.io/projected/0bf71631-d3f5-45d8-83fe-9702eff5cdf6-kube-api-access-hqjp5\") pod \"test-curl-7057-292d8a\" (UID: \"0bf71631-d3f5-45d8-83fe-9702eff5cdf6\") " pod="opendatahub/test-curl-7057-292d8a"
Oct 30 21:57:54.991412 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:54.986570    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fs5vb\""
Oct 30 21:57:55.029445 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:55.029407    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqjp5\" (UniqueName: \"kubernetes.io/projected/0bf71631-d3f5-45d8-83fe-9702eff5cdf6-kube-api-access-hqjp5\") pod \"test-curl-7057-292d8a\" (UID: \"0bf71631-d3f5-45d8-83fe-9702eff5cdf6\") " pod="opendatahub/test-curl-7057-292d8a"
Oct 30 21:57:55.090415 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:55.090290    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqjp5\" (UniqueName: \"kubernetes.io/projected/0bf71631-d3f5-45d8-83fe-9702eff5cdf6-kube-api-access-hqjp5\") pod \"test-curl-7057-292d8a\" (UID: \"0bf71631-d3f5-45d8-83fe-9702eff5cdf6\") " pod="opendatahub/test-curl-7057-292d8a"
Oct 30 21:57:55.203019 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:55.202965    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7057-292d8a"
Oct 30 21:57:55.589129 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:55.589095    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7057-292d8a"]
Oct 30 21:57:55.594075 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:55.594058    5087 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:57:55.789656 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:55.789617    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused" start-of-body=
Oct 30 21:57:55.789809 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:55.789689    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg" podUID="894d8554-651d-4ad9-8521-da6c98edca6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.133:15021/healthz/ready\": dial tcp 10.132.0.133:15021: connect: connection refused"
Oct 30 21:57:55.830087 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:55.830053    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body=
Oct 30 21:57:55.830271 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:55.830108    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k" podUID="89d4443a-0fb1-461e-a866-05351817130e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused"
Oct 30 21:57:55.851107 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:55.851026    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7057-292d8a" event={"ID":"0bf71631-d3f5-45d8-83fe-9702eff5cdf6","Type":"ContainerStarted","Data":"7b6848bb5746df10584077e146747f5a74c26f48284f19722411dfc5b08009f5"}
Oct 30 21:57:56.795652 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:56.795613    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:56.796629 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:56.796556    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:56.797575 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:56.797517    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 21:57:56.836051 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:56.836022    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:56.836478 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:56.836456    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:56.837125 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:56.837109    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 21:57:58.883631 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:58.883589    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7057-292d8a" event={"ID":"0bf71631-d3f5-45d8-83fe-9702eff5cdf6","Type":"ContainerStarted","Data":"dbbf81c7a50415ad7435044fdc0ae78e7ae9c85d15ce26628484bcd2c90787eb"}
Oct 30 21:57:59.000530 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:57:59.000474    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7057-292d8a" podStartSLOduration=2.186758432 podStartE2EDuration="5.000459005s" totalImagesPullingTime="2.813700573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:58.995567082 +0000 UTC m=+1473.470071060" watchObservedRunningTime="2026-10-30 21:57:59.000459005 +0000 UTC m=+1473.474962984"
Oct 30 21:58:07.474434 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:07.474357    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdd577bb-pjk4b"]
Oct 30 21:58:07.479186 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:07.479164    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd577bb-pjk4b"
Oct 30 21:58:07.535057 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:07.535035    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:58:07.542699 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:07.542674    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdd577bb-pjk4b"]
Oct 30 21:58:07.629720 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:07.629689    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f9bcb50-f502-4e1b-8d09-9d607752ac38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdd577bb-pjk4b\" (UID: \"3f9bcb50-f502-4e1b-8d09-9d607752ac38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd577bb-pjk4b"
Oct 30 21:58:07.629720 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:07.629732    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw492\" (UniqueName: \"kubernetes.io/projected/3f9bcb50-f502-4e1b-8d09-9d607752ac38-kube-api-access-fw492\") pod \"maas-api-revoke-sub-fdd577bb-pjk4b\" (UID: \"3f9bcb50-f502-4e1b-8d09-9d607752ac38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd577bb-pjk4b"
Oct 30 21:58:07.629928 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:07.629868    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f9bcb50-f502-4e1b-8d09-9d607752ac38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdd577bb-pjk4b\" (UID: \"3f9bcb50-f502-4e1b-8d09-9d607752ac38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd577bb-pjk4b"
Oct 30 21:58:07.630395 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:07.630376    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:58:07.731711 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:07.731676    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f9bcb50-f502-4e1b-8d09-9d607752ac38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdd577bb-pjk4b\" (UID: \"3f9bcb50-f502-4e1b-8d09-9d607752ac38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd577bb-pjk4b"
Oct 30 21:58:07.731884 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:07.731716    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw492\" (UniqueName: \"kubernetes.io/projected/3f9bcb50-f502-4e1b-8d09-9d607752ac38-kube-api-access-fw492\") pod \"maas-api-revoke-sub-fdd577bb-pjk4b\" (UID: \"3f9bcb50-f502-4e1b-8d09-9d607752ac38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd577bb-pjk4b"
Oct 30 21:58:07.731884 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:07.731779    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f9bcb50-f502-4e1b-8d09-9d607752ac38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdd577bb-pjk4b\" (UID: \"3f9bcb50-f502-4e1b-8d09-9d607752ac38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd577bb-pjk4b"
Oct 30 21:58:07.732509 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:07.732488    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f9bcb50-f502-4e1b-8d09-9d607752ac38-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdd577bb-pjk4b\" (UID: \"3f9bcb50-f502-4e1b-8d09-9d607752ac38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd577bb-pjk4b"
Oct 30 21:58:07.734177 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:07.734153    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f9bcb50-f502-4e1b-8d09-9d607752ac38-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdd577bb-pjk4b\" (UID: \"3f9bcb50-f502-4e1b-8d09-9d607752ac38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd577bb-pjk4b"
Oct 30 21:58:07.789027 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:07.789003    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw492\" (UniqueName: \"kubernetes.io/projected/3f9bcb50-f502-4e1b-8d09-9d607752ac38-kube-api-access-fw492\") pod \"maas-api-revoke-sub-fdd577bb-pjk4b\" (UID: \"3f9bcb50-f502-4e1b-8d09-9d607752ac38\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd577bb-pjk4b"
Oct 30 21:58:07.793168 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:07.793147    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd577bb-pjk4b"
Oct 30 21:58:08.182727 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:58:08.182686    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f9bcb50_f502_4e1b_8d09_9d607752ac38.slice/crio-035a7b5853eef8de4dab2e8fa11260cbddcf8ebea6d19083a6e9256c1bbdbdde WatchSource:0}: Error finding container 035a7b5853eef8de4dab2e8fa11260cbddcf8ebea6d19083a6e9256c1bbdbdde: Status 404 returned error can't find the container with id 035a7b5853eef8de4dab2e8fa11260cbddcf8ebea6d19083a6e9256c1bbdbdde
Oct 30 21:58:08.183390 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:08.183337    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdd577bb-pjk4b"]
Oct 30 21:58:08.969126 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:08.969081    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd577bb-pjk4b" event={"ID":"3f9bcb50-f502-4e1b-8d09-9d607752ac38","Type":"ContainerStarted","Data":"bf14abaffed3d1421e2feb0d542c2e50f6ea49d211407418327b73fc64de896e"}
Oct 30 21:58:08.969559 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:08.969141    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd577bb-pjk4b" event={"ID":"3f9bcb50-f502-4e1b-8d09-9d607752ac38","Type":"ContainerStarted","Data":"035a7b5853eef8de4dab2e8fa11260cbddcf8ebea6d19083a6e9256c1bbdbdde"}
Oct 30 21:58:09.091118 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:09.091067    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd577bb-pjk4b" podStartSLOduration=2.091053197 podStartE2EDuration="2.091053197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:09.089312553 +0000 UTC m=+1483.563816532" watchObservedRunningTime="2026-10-30 21:58:09.091053197 +0000 UTC m=+1483.565557189"
Oct 30 21:58:09.979079 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:09.979026    5087 generic.go:356] "Generic (PLEG): container finished" podID="3f9bcb50-f502-4e1b-8d09-9d607752ac38" containerID="bf14abaffed3d1421e2feb0d542c2e50f6ea49d211407418327b73fc64de896e" exitCode=0
Oct 30 21:58:09.979519 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:09.979076    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd577bb-pjk4b" event={"ID":"3f9bcb50-f502-4e1b-8d09-9d607752ac38","Type":"ContainerDied","Data":"bf14abaffed3d1421e2feb0d542c2e50f6ea49d211407418327b73fc64de896e"}
Oct 30 21:58:10.437121 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:10.435874    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7057-292d8a"]
Oct 30 21:58:10.437121 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:10.436132    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7057-292d8a" podUID="0bf71631-d3f5-45d8-83fe-9702eff5cdf6" containerName="test-curl-7057-292d8a" containerID="cri-o://dbbf81c7a50415ad7435044fdc0ae78e7ae9c85d15ce26628484bcd2c90787eb" gracePeriod=2
Oct 30 21:58:10.441961 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:10.441931    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7057-292d8a"]
Oct 30 21:58:10.488666 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:58:10.488621    5087 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7057-292d8a\" is forbidden: User \"system:node:ip-10-0-132-92.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-92.us-west-1.compute.internal' and this object" podUID="0bf71631-d3f5-45d8-83fe-9702eff5cdf6" pod="opendatahub/test-curl-7057-292d8a"
Oct 30 21:58:11.151119 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:11.151098    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd577bb-pjk4b"
Oct 30 21:58:11.181232 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:11.181206    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f9bcb50-f502-4e1b-8d09-9d607752ac38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f9bcb50-f502-4e1b-8d09-9d607752ac38-maas-api-openshift-ca\") pod \"3f9bcb50-f502-4e1b-8d09-9d607752ac38\" (UID: \"3f9bcb50-f502-4e1b-8d09-9d607752ac38\") "
Oct 30 21:58:11.181395 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:11.181279    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f9bcb50-f502-4e1b-8d09-9d607752ac38-kube-api-access-fw492\" (UniqueName: \"kubernetes.io/projected/3f9bcb50-f502-4e1b-8d09-9d607752ac38-kube-api-access-fw492\") pod \"3f9bcb50-f502-4e1b-8d09-9d607752ac38\" (UID: \"3f9bcb50-f502-4e1b-8d09-9d607752ac38\") "
Oct 30 21:58:11.181475 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:11.181460    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f9bcb50-f502-4e1b-8d09-9d607752ac38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f9bcb50-f502-4e1b-8d09-9d607752ac38-maas-api-xks-ca\") pod \"3f9bcb50-f502-4e1b-8d09-9d607752ac38\" (UID: \"3f9bcb50-f502-4e1b-8d09-9d607752ac38\") "
Oct 30 21:58:11.181535 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:11.181514    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f9bcb50-f502-4e1b-8d09-9d607752ac38-maas-api-openshift-ca" pod "3f9bcb50-f502-4e1b-8d09-9d607752ac38" (UID: "3f9bcb50-f502-4e1b-8d09-9d607752ac38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:11.181917 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:11.181900    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f9bcb50-f502-4e1b-8d09-9d607752ac38-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:58:11.183505 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:11.183479    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f9bcb50-f502-4e1b-8d09-9d607752ac38-maas-api-xks-ca" pod "3f9bcb50-f502-4e1b-8d09-9d607752ac38" (UID: "3f9bcb50-f502-4e1b-8d09-9d607752ac38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:11.183591 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:11.183551    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f9bcb50-f502-4e1b-8d09-9d607752ac38-kube-api-access-fw492" pod "3f9bcb50-f502-4e1b-8d09-9d607752ac38" (UID: "3f9bcb50-f502-4e1b-8d09-9d607752ac38"). InnerVolumeSpecName "kube-api-access-fw492". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:11.283336 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:11.283262    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f9bcb50-f502-4e1b-8d09-9d607752ac38-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:58:11.283336 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:11.283289    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw492\" (UniqueName: \"kubernetes.io/projected/3f9bcb50-f502-4e1b-8d09-9d607752ac38-kube-api-access-fw492\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:58:11.329910 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:58:11.329872    5087 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7057-292d8a\" is forbidden: User \"system:node:ip-10-0-132-92.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-92.us-west-1.compute.internal' and this object" podUID="0bf71631-d3f5-45d8-83fe-9702eff5cdf6" pod="opendatahub/test-curl-7057-292d8a"
Oct 30 21:58:11.995411 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:11.995370    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd577bb-pjk4b"
Oct 30 21:58:11.995411 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:11.995398    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdd577bb-pjk4b" event={"ID":"3f9bcb50-f502-4e1b-8d09-9d607752ac38","Type":"ContainerDied","Data":"035a7b5853eef8de4dab2e8fa11260cbddcf8ebea6d19083a6e9256c1bbdbdde"}
Oct 30 21:58:11.995623 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:11.995452    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="035a7b5853eef8de4dab2e8fa11260cbddcf8ebea6d19083a6e9256c1bbdbdde"
Oct 30 21:58:12.726833 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:12.726811    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7057-292d8a"
Oct 30 21:58:12.806331 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:12.806301    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bf71631-d3f5-45d8-83fe-9702eff5cdf6-kube-api-access-hqjp5\" (UniqueName: \"kubernetes.io/projected/0bf71631-d3f5-45d8-83fe-9702eff5cdf6-kube-api-access-hqjp5\") pod \"0bf71631-d3f5-45d8-83fe-9702eff5cdf6\" (UID: \"0bf71631-d3f5-45d8-83fe-9702eff5cdf6\") "
Oct 30 21:58:12.809405 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:12.809337    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bf71631-d3f5-45d8-83fe-9702eff5cdf6-kube-api-access-hqjp5" pod "0bf71631-d3f5-45d8-83fe-9702eff5cdf6" (UID: "0bf71631-d3f5-45d8-83fe-9702eff5cdf6"). InnerVolumeSpecName "kube-api-access-hqjp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:12.908165 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:12.908130    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqjp5\" (UniqueName: \"kubernetes.io/projected/0bf71631-d3f5-45d8-83fe-9702eff5cdf6-kube-api-access-hqjp5\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:58:13.006544 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:13.006505    5087 generic.go:356] "Generic (PLEG): container finished" podID="0bf71631-d3f5-45d8-83fe-9702eff5cdf6" containerID="dbbf81c7a50415ad7435044fdc0ae78e7ae9c85d15ce26628484bcd2c90787eb" exitCode=137
Oct 30 21:58:13.006736 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:13.006550    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7057-292d8a"
Oct 30 21:58:13.006736 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:13.006585    5087 scope.go:122] "RemoveContainer" containerID="dbbf81c7a50415ad7435044fdc0ae78e7ae9c85d15ce26628484bcd2c90787eb"
Oct 30 21:58:13.017933 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:13.017914    5087 scope.go:122] "RemoveContainer" containerID="dbbf81c7a50415ad7435044fdc0ae78e7ae9c85d15ce26628484bcd2c90787eb"
Oct 30 21:58:13.018190 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:58:13.018171    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbbf81c7a50415ad7435044fdc0ae78e7ae9c85d15ce26628484bcd2c90787eb\": container with ID starting with dbbf81c7a50415ad7435044fdc0ae78e7ae9c85d15ce26628484bcd2c90787eb not found: ID does not exist" containerID="dbbf81c7a50415ad7435044fdc0ae78e7ae9c85d15ce26628484bcd2c90787eb"
Oct 30 21:58:13.018226 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:13.018200    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbbf81c7a50415ad7435044fdc0ae78e7ae9c85d15ce26628484bcd2c90787eb"} err="failed to get container status \"dbbf81c7a50415ad7435044fdc0ae78e7ae9c85d15ce26628484bcd2c90787eb\": rpc error: code = NotFound desc = could not find container \"dbbf81c7a50415ad7435044fdc0ae78e7ae9c85d15ce26628484bcd2c90787eb\": container with ID starting with dbbf81c7a50415ad7435044fdc0ae78e7ae9c85d15ce26628484bcd2c90787eb not found: ID does not exist"
Oct 30 21:58:13.066235 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:58:13.066190    5087 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7057-292d8a\" is forbidden: User \"system:node:ip-10-0-132-92.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-92.us-west-1.compute.internal' and this object" podUID="0bf71631-d3f5-45d8-83fe-9702eff5cdf6" pod="opendatahub/test-curl-7057-292d8a"
Oct 30 21:58:13.724990 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:13.724945    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bf71631-d3f5-45d8-83fe-9702eff5cdf6" path="/var/lib/kubelet/pods/0bf71631-d3f5-45d8-83fe-9702eff5cdf6/volumes"
Oct 30 21:58:16.911851 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:16.911817    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"]
Oct 30 21:58:16.914141 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:16.912482    5087 cpu_manager.go:414] "containerMap: removing container" podUID="0bf71631-d3f5-45d8-83fe-9702eff5cdf6" containerName="test-curl-7057-292d8a"
Oct 30 21:58:16.914141 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:16.912495    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bf71631-d3f5-45d8-83fe-9702eff5cdf6" containerName="test-curl-7057-292d8a"
Oct 30 21:58:16.914141 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:16.912524    5087 cpu_manager.go:414] "containerMap: removing container" podUID="3f9bcb50-f502-4e1b-8d09-9d607752ac38" containerName="revoke-keys"
Oct 30 21:58:16.914141 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:16.912531    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f9bcb50-f502-4e1b-8d09-9d607752ac38" containerName="revoke-keys"
Oct 30 21:58:16.914141 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:16.912628    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f9bcb50-f502-4e1b-8d09-9d607752ac38" containerName="revoke-keys"
Oct 30 21:58:16.914141 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:16.912642    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bf71631-d3f5-45d8-83fe-9702eff5cdf6" containerName="test-curl-7057-292d8a"
Oct 30 21:58:16.916058 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:16.916043    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:16.961023 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:16.960988    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"]
Oct 30 21:58:16.965457 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:16.965426    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:16.971825 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:16.971793    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-d5bcl\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:16.971942 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:16.971867    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-d5bcl\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:16.971983 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:16.971956    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-d5bcl\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:16.972018 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:16.971986    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-d5bcl\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:16.972018 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:16.972013    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-d5bcl\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:16.972165 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:16.972147    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-d5bcl\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:16.972430 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:16.972409    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8m2v\" (UniqueName: \"kubernetes.io/projected/061e2d92-1dd6-4373-b93a-09dd58d23a0e-kube-api-access-j8m2v\") pod \"maas-default-gateway-openshift-default-74854d8ccf-d5bcl\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:16.972501 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:16.972472    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-d5bcl\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:16.972561 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:16.972522    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-d5bcl\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:16.978529 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:16.978512    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-xn2np\""
Oct 30 21:58:17.018399 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.018368    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"]
Oct 30 21:58:17.070667 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.070631    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"]
Oct 30 21:58:17.073676 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.073648    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62fa8024-dac6-4993-b671-23bda513f309-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-xmvjc\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:17.073934 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.073702    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-xmvjc\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:17.073934 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.073745    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8m2v\" (UniqueName: \"kubernetes.io/projected/061e2d92-1dd6-4373-b93a-09dd58d23a0e-kube-api-access-j8m2v\") pod \"maas-default-gateway-openshift-default-74854d8ccf-d5bcl\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:17.073934 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.073838    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-d5bcl\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:17.073934 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.073886    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-d5bcl\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:17.073934 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.073910    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-xmvjc\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:17.073934 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.073932    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62fa8024-dac6-4993-b671-23bda513f309-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-xmvjc\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:17.074248 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.073993    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-xmvjc\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:17.074248 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.074092    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-d5bcl\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:17.074248 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.074138    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2lkq\" (UniqueName: \"kubernetes.io/projected/62fa8024-dac6-4993-b671-23bda513f309-kube-api-access-p2lkq\") pod \"maas-default-gateway-openshift-default-74854d8ccf-xmvjc\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:17.074248 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.074174    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-xmvjc\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:17.074248 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.074207    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-d5bcl\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:17.074535 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.074262    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-d5bcl\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:17.074535 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.074293    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-d5bcl\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:17.074535 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.074325    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-d5bcl\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:17.074535 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.074369    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62fa8024-dac6-4993-b671-23bda513f309-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-xmvjc\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:17.074535 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.074451    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-d5bcl\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:17.074535 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.074500    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-xmvjc\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:17.075538 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.075514    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-d5bcl\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:17.076988 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.076965    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-d5bcl\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:17.077114 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.077090    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-d5bcl\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:17.077182 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.077134    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-d5bcl\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:17.077626 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.077602    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-d5bcl\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:17.078644 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.078620    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-d5bcl\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:17.080585 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.080557    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-d5bcl\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:17.136164 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.136135    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8m2v\" (UniqueName: \"kubernetes.io/projected/061e2d92-1dd6-4373-b93a-09dd58d23a0e-kube-api-access-j8m2v\") pod \"maas-default-gateway-openshift-default-74854d8ccf-d5bcl\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:17.136732 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.136712    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-d5bcl\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:17.169191 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:58:17.169118    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:58:17.176012 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.175991    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-xmvjc\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:17.176172 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.176086    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62fa8024-dac6-4993-b671-23bda513f309-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-xmvjc\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:17.176172 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.176111    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-xmvjc\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:17.176300 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.176208    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-xmvjc\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:17.176300 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.176240    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62fa8024-dac6-4993-b671-23bda513f309-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-xmvjc\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:17.176436 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.176298    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-xmvjc\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:17.176436 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.176423    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2lkq\" (UniqueName: \"kubernetes.io/projected/62fa8024-dac6-4993-b671-23bda513f309-kube-api-access-p2lkq\") pod \"maas-default-gateway-openshift-default-74854d8ccf-xmvjc\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:17.176535 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.176431    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-xmvjc\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:17.176535 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.176457    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-xmvjc\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:17.176772 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.176533    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62fa8024-dac6-4993-b671-23bda513f309-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-xmvjc\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:17.176772 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.176530    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-xmvjc\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:17.176977 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.176952    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-xmvjc\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:17.177315 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.177004    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-xmvjc\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:17.177315 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.177279    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62fa8024-dac6-4993-b671-23bda513f309-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-xmvjc\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:17.178984 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.178964    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62fa8024-dac6-4993-b671-23bda513f309-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-xmvjc\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:17.179618 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.179600    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-xmvjc\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:17.230269 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.230238    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:17.234965 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.234946    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2lkq\" (UniqueName: \"kubernetes.io/projected/62fa8024-dac6-4993-b671-23bda513f309-kube-api-access-p2lkq\") pod \"maas-default-gateway-openshift-default-74854d8ccf-xmvjc\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:17.239806 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.239782    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62fa8024-dac6-4993-b671-23bda513f309-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-xmvjc\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:17.280428 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.279860    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:17.685708 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.685675    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"]
Oct 30 21:58:17.717175 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:58:17.717135    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061e2d92_1dd6_4373_b93a_09dd58d23a0e.slice/crio-c926c7eec59b91be0a776214529426ca856ee771e279885f4629f93454766eab WatchSource:0}: Error finding container c926c7eec59b91be0a776214529426ca856ee771e279885f4629f93454766eab: Status 404 returned error can't find the container with id c926c7eec59b91be0a776214529426ca856ee771e279885f4629f93454766eab
Oct 30 21:58:17.719805 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.719775    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:58:17.719896 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.719862    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:58:17.719896 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.719893    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:58:17.744913 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.744887    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"]
Oct 30 21:58:17.745947 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.745909    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:58:17.746049 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.745993    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:58:17.746049 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:17.746032    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:58:18.051342 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:18.051305    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc" event={"ID":"62fa8024-dac6-4993-b671-23bda513f309","Type":"ContainerStarted","Data":"be67215a5cbffa912d1a3de9277cf14d96e9435b5c1ebe523d7461be348c7bcb"}
Oct 30 21:58:18.051921 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:18.051423    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc" event={"ID":"62fa8024-dac6-4993-b671-23bda513f309","Type":"ContainerStarted","Data":"0ebc3d1369780cba4d624673c9405f5e7eb8fcee4e2b987e18a100db0c5e6837"}
Oct 30 21:58:18.053682 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:18.053651    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl" event={"ID":"061e2d92-1dd6-4373-b93a-09dd58d23a0e","Type":"ContainerStarted","Data":"94e5dfb2d66b0be6a057524b68d8131c4521f2cc668683d3cdcc993e11ff2650"}
Oct 30 21:58:18.053840 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:18.053713    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl" event={"ID":"061e2d92-1dd6-4373-b93a-09dd58d23a0e","Type":"ContainerStarted","Data":"c926c7eec59b91be0a776214529426ca856ee771e279885f4629f93454766eab"}
Oct 30 21:58:18.178915 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:18.178864    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc" podStartSLOduration=2.178850706 podStartE2EDuration="2.178850706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:18.172679456 +0000 UTC m=+1492.647183436" watchObservedRunningTime="2026-10-30 21:58:18.178850706 +0000 UTC m=+1492.653354684"
Oct 30 21:58:18.230874 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:18.230840    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:18.232818 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:18.232791    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-d5bcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 30 21:58:18.232953 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:18.232842    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl" podUID="061e2d92-1dd6-4373-b93a-09dd58d23a0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 30 21:58:18.281342 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:18.281296    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:18.283431 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:18.283404    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-xmvjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 21:58:18.283557 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:18.283463    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc" podUID="62fa8024-dac6-4993-b671-23bda513f309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 21:58:18.299612 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:18.299545    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl" podStartSLOduration=2.29952334 podStartE2EDuration="2.29952334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:18.289707735 +0000 UTC m=+1492.764211714" watchObservedRunningTime="2026-10-30 21:58:18.29952334 +0000 UTC m=+1492.774027321"
Oct 30 21:58:19.231208 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:19.231175    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-d5bcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 30 21:58:19.231647 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:19.231236    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl" podUID="061e2d92-1dd6-4373-b93a-09dd58d23a0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 30 21:58:19.280862 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:19.280828    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-xmvjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 21:58:19.281145 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:19.281117    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc" podUID="62fa8024-dac6-4993-b671-23bda513f309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 21:58:20.231758 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:20.231723    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-d5bcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 30 21:58:20.232376 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:20.232322    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl" podUID="061e2d92-1dd6-4373-b93a-09dd58d23a0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 30 21:58:20.281377 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:20.280769    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-xmvjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 21:58:20.281377 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:20.280833    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc" podUID="62fa8024-dac6-4993-b671-23bda513f309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 21:58:21.230571 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:21.230534    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-d5bcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 30 21:58:21.230571 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:21.230605    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl" podUID="061e2d92-1dd6-4373-b93a-09dd58d23a0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 30 21:58:21.281768 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:21.281728    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-xmvjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 21:58:21.282385 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:21.282330    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc" podUID="62fa8024-dac6-4993-b671-23bda513f309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 21:58:22.231335 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:22.231300    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-d5bcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 30 21:58:22.231542 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:22.231366    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl" podUID="061e2d92-1dd6-4373-b93a-09dd58d23a0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 30 21:58:22.281372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:22.280755    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-xmvjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 21:58:22.281372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:22.280818    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc" podUID="62fa8024-dac6-4993-b671-23bda513f309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 21:58:23.231661 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:23.231624    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-d5bcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 30 21:58:23.232164 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:23.231684    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl" podUID="061e2d92-1dd6-4373-b93a-09dd58d23a0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 30 21:58:23.281374 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:23.280710    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-xmvjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 21:58:23.281374 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:23.280784    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc" podUID="62fa8024-dac6-4993-b671-23bda513f309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 21:58:24.232054 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:24.232016    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-d5bcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 30 21:58:24.232578 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:24.232076    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl" podUID="061e2d92-1dd6-4373-b93a-09dd58d23a0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 30 21:58:24.281207 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:24.281158    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-xmvjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 21:58:24.281461 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:24.281231    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc" podUID="62fa8024-dac6-4993-b671-23bda513f309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 21:58:25.230727 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:25.230685    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-d5bcl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 30 21:58:25.230950 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:25.230753    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl" podUID="061e2d92-1dd6-4373-b93a-09dd58d23a0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 30 21:58:25.280769 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:25.280735    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-xmvjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused" start-of-body=
Oct 30 21:58:25.281260 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:25.280784    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc" podUID="62fa8024-dac6-4993-b671-23bda513f309" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.138:15021/healthz/ready\": dial tcp 10.132.0.138:15021: connect: connection refused"
Oct 30 21:58:25.933113 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:25.933080    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 21:58:25.943083 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:25.943028    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 21:58:25.947912 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:25.947885    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 21:58:25.957481 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:25.957444    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 21:58:26.923058 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:26.923025    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:26.923821 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:26.923801    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:26.924741 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:26.924718    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 21:58:26.944849 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:26.944817    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:26.946292 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:26.946191    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:26.947896 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:26.947874    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 21:58:27.636786 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:27.636746    5087 scope.go:122] "RemoveContainer" containerID="1bb762b202bd7d179192284710ab1142fea012d136cbf7dfcc3bbbc2fd1e86ce"
Oct 30 21:58:39.110373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:39.095235    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:58:39.441452 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:39.430447    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:58:39.441452 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:39.431069    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:58:39.573476 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:39.571465    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:58:39.573476 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:39.572491    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded"
Oct 30 21:58:39.580550 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:39.580505    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:58:39.652197 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:39.651264    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:58:39.981374 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:39.972447    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:58:39.981374 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:39.972534    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96" podUID="727b252d-ad3e-4e43-9fb8-471fd2957231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:58:47.172903 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:58:47.172862    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:58:51.062199 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:51.062167    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c04fcec-4dng2"]
Oct 30 21:58:51.110923 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:51.110890    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c04fcec-4dng2"]
Oct 30 21:58:51.711926 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:58:51.711891    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20fc6591-9453-4da6-977b-e30509b13264" path="/var/lib/kubelet/pods/20fc6591-9453-4da6-977b-e30509b13264/volumes"
Oct 30 21:59:02.363286 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.363254    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"]
Oct 30 21:59:02.369486 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.369465    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:02.411965 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.411939    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"]
Oct 30 21:59:02.417110 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.417089    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:02.465717 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.465682    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"]
Oct 30 21:59:02.502813 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.502783    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"]
Oct 30 21:59:02.510975 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.510949    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-jj2hd\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:02.511131 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.510989    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-jj2hd\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:02.511131 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.511020    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8bss\" (UniqueName: \"kubernetes.io/projected/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-kube-api-access-d8bss\") pod \"maas-default-gateway-openshift-default-74854d8ccf-jj2hd\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:02.511131 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.511042    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-fskjv\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:02.511131 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.511066    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc565097-2279-4f89-a2a6-35e117fd929a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-fskjv\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:02.511131 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.511110    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-jj2hd\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:02.511510 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.511153    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-fskjv\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:02.511510 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.511216    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljbv6\" (UniqueName: \"kubernetes.io/projected/bc565097-2279-4f89-a2a6-35e117fd929a-kube-api-access-ljbv6\") pod \"maas-default-gateway-openshift-default-74854d8ccf-fskjv\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:02.511510 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.511266    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-jj2hd\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:02.511510 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.511296    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-jj2hd\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:02.511510 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.511325    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-fskjv\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:02.511510 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.511395    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc565097-2279-4f89-a2a6-35e117fd929a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-fskjv\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:02.511510 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.511458    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-jj2hd\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:02.511862 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.511547    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-fskjv\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:02.511862 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.511621    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-fskjv\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:02.511862 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.511651    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-jj2hd\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:02.511862 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.511742    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-jj2hd\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:02.511862 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.511776    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc565097-2279-4f89-a2a6-35e117fd929a-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-fskjv\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:02.612911 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.612878    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-jj2hd\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:02.613077 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.612920    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc565097-2279-4f89-a2a6-35e117fd929a-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-fskjv\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:02.613077 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.613033    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-jj2hd\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:02.613077 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.613066    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-jj2hd\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:02.613278 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.613118    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8bss\" (UniqueName: \"kubernetes.io/projected/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-kube-api-access-d8bss\") pod \"maas-default-gateway-openshift-default-74854d8ccf-jj2hd\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:02.613456 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.613145    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-fskjv\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:02.613661 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.613630    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc565097-2279-4f89-a2a6-35e117fd929a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-fskjv\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:02.613805 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.613706    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-jj2hd\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:02.613805 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.613707    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-fskjv\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:02.613805 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.613774    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-fskjv\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:02.613945 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.613821    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-jj2hd\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:02.613945 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.613847    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljbv6\" (UniqueName: \"kubernetes.io/projected/bc565097-2279-4f89-a2a6-35e117fd929a-kube-api-access-ljbv6\") pod \"maas-default-gateway-openshift-default-74854d8ccf-fskjv\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:02.613945 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.613911    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-jj2hd\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:02.614051 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.613944    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-jj2hd\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:02.614051 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.613988    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-fskjv\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:02.614051 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.614012    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc565097-2279-4f89-a2a6-35e117fd929a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-fskjv\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:02.614232 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.614066    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-jj2hd\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:02.614232 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.614075    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-jj2hd\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:02.614232 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.614129    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-fskjv\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:02.614232 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.614150    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-fskjv\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:02.614232 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.614208    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-fskjv\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:02.614768 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.614239    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-jj2hd\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:02.614768 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.614492    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-jj2hd\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:02.615425 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.614997    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-jj2hd\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:02.615425 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.615020    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-jj2hd\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:02.615425 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.615142    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-fskjv\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:02.615425 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.615335    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-fskjv\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:02.615722 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.615505    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc565097-2279-4f89-a2a6-35e117fd929a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-fskjv\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:02.616733 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.616702    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-fskjv\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:02.616851 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.616777    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-jj2hd\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:02.616952 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.616937    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc565097-2279-4f89-a2a6-35e117fd929a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-fskjv\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:02.617000 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.616989    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-jj2hd\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:02.690657 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.690626    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-jj2hd\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:02.691224 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.691190    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc565097-2279-4f89-a2a6-35e117fd929a-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-fskjv\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:02.692811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.692775    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8bss\" (UniqueName: \"kubernetes.io/projected/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-kube-api-access-d8bss\") pod \"maas-default-gateway-openshift-default-74854d8ccf-jj2hd\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:02.693261 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.693236    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljbv6\" (UniqueName: \"kubernetes.io/projected/bc565097-2279-4f89-a2a6-35e117fd929a-kube-api-access-ljbv6\") pod \"maas-default-gateway-openshift-default-74854d8ccf-fskjv\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:02.732647 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.732619    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:02.983818 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:02.983786    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:03.186815 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:59:03.186785    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod280b7c38_6ad1_4e93_ba46_ba4ae8171ca9.slice/crio-6560fdfb1806473855ae1c1cc27c6482c28a7bf80297aaa95e076c5cabfeedc1 WatchSource:0}: Error finding container 6560fdfb1806473855ae1c1cc27c6482c28a7bf80297aaa95e076c5cabfeedc1: Status 404 returned error can't find the container with id 6560fdfb1806473855ae1c1cc27c6482c28a7bf80297aaa95e076c5cabfeedc1
Oct 30 21:59:03.187073 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:03.187052    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"]
Oct 30 21:59:03.189811 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:03.189782    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:59:03.189882 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:03.189844    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:59:03.189882 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:03.189873    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:59:03.386784 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:59:03.386754    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc565097_2279_4f89_a2a6_35e117fd929a.slice/crio-5b7265d793a30a2c0fe17ba06997df5d08f2547feb6a6b4464941be571353285 WatchSource:0}: Error finding container 5b7265d793a30a2c0fe17ba06997df5d08f2547feb6a6b4464941be571353285: Status 404 returned error can't find the container with id 5b7265d793a30a2c0fe17ba06997df5d08f2547feb6a6b4464941be571353285
Oct 30 21:59:03.387735 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:03.387713    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"]
Oct 30 21:59:03.389548 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:03.389513    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:59:03.389663 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:03.389591    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:59:03.389663 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:03.389634    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:59:04.118040 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:04.118005    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv" event={"ID":"bc565097-2279-4f89-a2a6-35e117fd929a","Type":"ContainerStarted","Data":"3cc130c3ff474265a6b8dd8c8353b5ba573d5d7ed8469e2b235d7b9ba8beefa9"}
Oct 30 21:59:04.118214 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:04.118099    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv" event={"ID":"bc565097-2279-4f89-a2a6-35e117fd929a","Type":"ContainerStarted","Data":"5b7265d793a30a2c0fe17ba06997df5d08f2547feb6a6b4464941be571353285"}
Oct 30 21:59:04.120597 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:04.119937    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd" event={"ID":"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9","Type":"ContainerStarted","Data":"fb3deb51b4b87b4e24ff09d3aab21c372977abc28aec7479efca379e39630308"}
Oct 30 21:59:04.120597 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:04.119980    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd" event={"ID":"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9","Type":"ContainerStarted","Data":"6560fdfb1806473855ae1c1cc27c6482c28a7bf80297aaa95e076c5cabfeedc1"}
Oct 30 21:59:04.494297 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:04.491507    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv" podStartSLOduration=2.4914847829999998 podStartE2EDuration="2.491484783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:04.349667538 +0000 UTC m=+1538.824171518" watchObservedRunningTime="2026-10-30 21:59:04.491484783 +0000 UTC m=+1538.965988762"
Oct 30 21:59:04.732898 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:04.732870    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:04.734699 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:04.734664    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-jj2hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 21:59:04.734834 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:04.734731    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd" podUID="280b7c38-6ad1-4e93-ba46-ba4ae8171ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 21:59:04.985785 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:04.985695    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:04.997393 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:04.996173    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-fskjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 21:59:04.997393 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:04.996266    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv" podUID="bc565097-2279-4f89-a2a6-35e117fd929a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 21:59:05.733242 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:05.733210    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-jj2hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 21:59:05.733857 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:05.733824    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd" podUID="280b7c38-6ad1-4e93-ba46-ba4ae8171ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 21:59:05.984968 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:05.984453    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-fskjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 21:59:05.984968 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:05.984510    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv" podUID="bc565097-2279-4f89-a2a6-35e117fd929a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 21:59:06.733185 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:06.733139    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-jj2hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 21:59:06.733382 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:06.733200    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd" podUID="280b7c38-6ad1-4e93-ba46-ba4ae8171ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 21:59:06.991437 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:06.990408    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-fskjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 21:59:06.991437 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:06.990473    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv" podUID="bc565097-2279-4f89-a2a6-35e117fd929a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 21:59:07.733520 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:07.733485    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-jj2hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 21:59:07.734020 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:07.733566    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd" podUID="280b7c38-6ad1-4e93-ba46-ba4ae8171ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 21:59:07.989430 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:07.989080    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-fskjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 21:59:07.989430 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:07.989154    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv" podUID="bc565097-2279-4f89-a2a6-35e117fd929a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 21:59:08.733395 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:08.733360    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-jj2hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 21:59:08.733592 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:08.733420    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd" podUID="280b7c38-6ad1-4e93-ba46-ba4ae8171ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 21:59:08.991252 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:08.988280    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-fskjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 21:59:08.991252 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:08.988375    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv" podUID="bc565097-2279-4f89-a2a6-35e117fd929a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 21:59:09.733765 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:09.733730    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-jj2hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 21:59:09.734254 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:09.733792    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd" podUID="280b7c38-6ad1-4e93-ba46-ba4ae8171ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 21:59:09.984517 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:09.984417    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-fskjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 21:59:09.984517 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:09.984501    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv" podUID="bc565097-2279-4f89-a2a6-35e117fd929a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 21:59:10.733550 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:10.733515    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-jj2hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 30 21:59:10.733761 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:10.733576    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd" podUID="280b7c38-6ad1-4e93-ba46-ba4ae8171ca9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 30 21:59:10.984791 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:10.984706    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-fskjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body=
Oct 30 21:59:10.985193 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:10.984792    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv" podUID="bc565097-2279-4f89-a2a6-35e117fd929a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused"
Oct 30 21:59:12.417584 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:12.417393    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:12.418616 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:12.418592    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:12.422220 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:12.422198    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 21:59:12.643498 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:12.643432    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd" podStartSLOduration=10.643415471 podStartE2EDuration="10.643415471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:04.480679097 +0000 UTC m=+1538.955183079" watchObservedRunningTime="2026-10-30 21:59:12.643415471 +0000 UTC m=+1547.117919452"
Oct 30 21:59:12.684426 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:12.683903    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:12.684687 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:12.684664    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:12.685740 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:12.685717    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 21:59:17.169959 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:59:17.169922    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:59:23.612871 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.612834    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"]
Oct 30 21:59:23.617944 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.617921    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:23.679201 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.679177    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e585c53f-openshift-default-dockercfg-svzf8\""
Oct 30 21:59:23.708235 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.708209    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-credential-socket\") pod \"e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") " pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:23.708384 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.708259    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f71bde2-938d-4994-8a4e-6f37bbbee412-istio-token\") pod \"e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") " pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:23.708384 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.708358    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-istio-envoy\") pod \"e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") " pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:23.708517 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.708402    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f71bde2-938d-4994-8a4e-6f37bbbee412-istiod-ca-cert\") pod \"e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") " pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:23.708595 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.708576    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d4rp\" (UniqueName: \"kubernetes.io/projected/3f71bde2-938d-4994-8a4e-6f37bbbee412-kube-api-access-9d4rp\") pod \"e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") " pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:23.708661 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.708614    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f71bde2-938d-4994-8a4e-6f37bbbee412-istio-podinfo\") pod \"e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") " pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:23.708729 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.708673    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-istio-data\") pod \"e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") " pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:23.708729 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.708719    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-workload-certs\") pod \"e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") " pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:23.708862 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.708849    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-workload-socket\") pod \"e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") " pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:23.742126 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.742092    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"]
Oct 30 21:59:23.810798 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.810765    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-credential-socket\") pod \"e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") " pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:23.810982 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.810819    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f71bde2-938d-4994-8a4e-6f37bbbee412-istio-token\") pod \"e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") " pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:23.810982 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.810848    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-istio-envoy\") pod \"e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") " pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:23.810982 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.810881    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f71bde2-938d-4994-8a4e-6f37bbbee412-istiod-ca-cert\") pod \"e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") " pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:23.810982 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.810949    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d4rp\" (UniqueName: \"kubernetes.io/projected/3f71bde2-938d-4994-8a4e-6f37bbbee412-kube-api-access-9d4rp\") pod \"e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") " pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:23.810982 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.810967    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f71bde2-938d-4994-8a4e-6f37bbbee412-istio-podinfo\") pod \"e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") " pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:23.811270 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.810985    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-istio-data\") pod \"e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") " pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:23.811270 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.811152    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-workload-certs\") pod \"e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") " pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:23.811270 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.811157    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-credential-socket\") pod \"e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") " pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:23.811419 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.811280    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-workload-socket\") pod \"e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") " pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:23.811419 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.811335    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-istio-data\") pod \"e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") " pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:23.811535 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.811432    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-workload-certs\") pod \"e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") " pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:23.811619 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.811602    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-workload-socket\") pod \"e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") " pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:23.811760 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.811733    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f71bde2-938d-4994-8a4e-6f37bbbee412-istiod-ca-cert\") pod \"e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") " pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:23.813191 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.813168    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-istio-envoy\") pod \"e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") " pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:23.813488 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.813470    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f71bde2-938d-4994-8a4e-6f37bbbee412-istio-podinfo\") pod \"e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") " pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:23.870904 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.870824    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f71bde2-938d-4994-8a4e-6f37bbbee412-istio-token\") pod \"e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") " pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:23.872019 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.871990    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d4rp\" (UniqueName: \"kubernetes.io/projected/3f71bde2-938d-4994-8a4e-6f37bbbee412-kube-api-access-9d4rp\") pod \"e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") " pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:23.933598 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:23.933568    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:24.439738 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:59:24.439705    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f71bde2_938d_4994_8a4e_6f37bbbee412.slice/crio-7c6775d1026fffc5ddf8dea4b4f0785bc332adf33ca021e86e0cd2ce034a44f2 WatchSource:0}: Error finding container 7c6775d1026fffc5ddf8dea4b4f0785bc332adf33ca021e86e0cd2ce034a44f2: Status 404 returned error can't find the container with id 7c6775d1026fffc5ddf8dea4b4f0785bc332adf33ca021e86e0cd2ce034a44f2
Oct 30 21:59:24.440790 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:24.440765    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"]
Oct 30 21:59:24.443531 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:24.443493    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:59:24.443652 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:24.443583    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:59:24.443652 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:24.443630    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 21:59:25.370890 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:25.370852    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd" event={"ID":"3f71bde2-938d-4994-8a4e-6f37bbbee412","Type":"ContainerStarted","Data":"8f86807b09a71d7b481cb4f47603445bf891eab4df9022dd164870c32144998f"}
Oct 30 21:59:25.371294 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:25.370899    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd" event={"ID":"3f71bde2-938d-4994-8a4e-6f37bbbee412","Type":"ContainerStarted","Data":"7c6775d1026fffc5ddf8dea4b4f0785bc332adf33ca021e86e0cd2ce034a44f2"}
Oct 30 21:59:25.598428 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:25.598327    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd" podStartSLOduration=2.598309828 podStartE2EDuration="2.598309828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:25.547837433 +0000 UTC m=+1560.022341436" watchObservedRunningTime="2026-10-30 21:59:25.598309828 +0000 UTC m=+1560.072813808"
Oct 30 21:59:25.934191 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:25.934161    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:25.935578 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:25.935555    5087 patch_prober.go:29] interesting pod/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 21:59:25.935707 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:25.935600    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd" podUID="3f71bde2-938d-4994-8a4e-6f37bbbee412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 21:59:26.183760 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:26.183721    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f05b6443-2mkvj"]
Oct 30 21:59:26.188559 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:26.188495    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f05b6443-2mkvj"
Oct 30 21:59:26.257372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:26.257165    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:59:26.257372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:26.257269    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:59:26.267620 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:26.265619    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f05b6443-2mkvj"]
Oct 30 21:59:26.280518 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:26.280479    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djbgw\" (UniqueName: \"kubernetes.io/projected/c2e34517-339f-4a98-8892-abbc0e6fb627-kube-api-access-djbgw\") pod \"maas-api-revoke-keys-e2e-ait-f05b6443-2mkvj\" (UID: \"c2e34517-339f-4a98-8892-abbc0e6fb627\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f05b6443-2mkvj"
Oct 30 21:59:26.280518 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:26.280514    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2e34517-339f-4a98-8892-abbc0e6fb627-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f05b6443-2mkvj\" (UID: \"c2e34517-339f-4a98-8892-abbc0e6fb627\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f05b6443-2mkvj"
Oct 30 21:59:26.280746 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:26.280709    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2e34517-339f-4a98-8892-abbc0e6fb627-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f05b6443-2mkvj\" (UID: \"c2e34517-339f-4a98-8892-abbc0e6fb627\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f05b6443-2mkvj"
Oct 30 21:59:26.382155 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:26.382125    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2e34517-339f-4a98-8892-abbc0e6fb627-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f05b6443-2mkvj\" (UID: \"c2e34517-339f-4a98-8892-abbc0e6fb627\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f05b6443-2mkvj"
Oct 30 21:59:26.382597 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:26.382391    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djbgw\" (UniqueName: \"kubernetes.io/projected/c2e34517-339f-4a98-8892-abbc0e6fb627-kube-api-access-djbgw\") pod \"maas-api-revoke-keys-e2e-ait-f05b6443-2mkvj\" (UID: \"c2e34517-339f-4a98-8892-abbc0e6fb627\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f05b6443-2mkvj"
Oct 30 21:59:26.382597 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:26.382427    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2e34517-339f-4a98-8892-abbc0e6fb627-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f05b6443-2mkvj\" (UID: \"c2e34517-339f-4a98-8892-abbc0e6fb627\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f05b6443-2mkvj"
Oct 30 21:59:26.383136 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:26.383118    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2e34517-339f-4a98-8892-abbc0e6fb627-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f05b6443-2mkvj\" (UID: \"c2e34517-339f-4a98-8892-abbc0e6fb627\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f05b6443-2mkvj"
Oct 30 21:59:26.384758 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:26.384735    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2e34517-339f-4a98-8892-abbc0e6fb627-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f05b6443-2mkvj\" (UID: \"c2e34517-339f-4a98-8892-abbc0e6fb627\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f05b6443-2mkvj"
Oct 30 21:59:26.464372 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:26.464280    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djbgw\" (UniqueName: \"kubernetes.io/projected/c2e34517-339f-4a98-8892-abbc0e6fb627-kube-api-access-djbgw\") pod \"maas-api-revoke-keys-e2e-ait-f05b6443-2mkvj\" (UID: \"c2e34517-339f-4a98-8892-abbc0e6fb627\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f05b6443-2mkvj"
Oct 30 21:59:26.502531 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:26.502501    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f05b6443-2mkvj"
Oct 30 21:59:26.928633 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:59:26.928605    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e34517_339f_4a98_8892_abbc0e6fb627.slice/crio-4b2be7966580c4b324ec5d7b0f2a152fe89a5015b7632397f20c6522bc2c99df WatchSource:0}: Error finding container 4b2be7966580c4b324ec5d7b0f2a152fe89a5015b7632397f20c6522bc2c99df: Status 404 returned error can't find the container with id 4b2be7966580c4b324ec5d7b0f2a152fe89a5015b7632397f20c6522bc2c99df
Oct 30 21:59:26.929285 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:26.929261    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f05b6443-2mkvj"]
Oct 30 21:59:26.935752 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:26.935714    5087 patch_prober.go:29] interesting pod/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 21:59:26.935862 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:26.935771    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd" podUID="3f71bde2-938d-4994-8a4e-6f37bbbee412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 21:59:27.389539 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:27.389508    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f05b6443-2mkvj" event={"ID":"c2e34517-339f-4a98-8892-abbc0e6fb627","Type":"ContainerStarted","Data":"4b2be7966580c4b324ec5d7b0f2a152fe89a5015b7632397f20c6522bc2c99df"}
Oct 30 21:59:27.933980 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:27.933930    5087 patch_prober.go:29] interesting pod/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 21:59:27.934187 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:27.933985    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd" podUID="3f71bde2-938d-4994-8a4e-6f37bbbee412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 21:59:28.398832 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:28.398795    5087 generic.go:356] "Generic (PLEG): container finished" podID="c2e34517-339f-4a98-8892-abbc0e6fb627" containerID="a2bb68712d151a05a5718090c69ea48179fefe49e51f449c14865ff61b58e52d" exitCode=0
Oct 30 21:59:28.399216 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:28.399075    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f05b6443-2mkvj" event={"ID":"c2e34517-339f-4a98-8892-abbc0e6fb627","Type":"ContainerDied","Data":"a2bb68712d151a05a5718090c69ea48179fefe49e51f449c14865ff61b58e52d"}
Oct 30 21:59:28.934505 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:28.934471    5087 patch_prober.go:29] interesting pod/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 21:59:28.934707 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:28.934520    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd" podUID="3f71bde2-938d-4994-8a4e-6f37bbbee412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 21:59:29.727182 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:29.727155    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f05b6443-2mkvj"
Oct 30 21:59:29.860373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:29.856979    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2e34517-339f-4a98-8892-abbc0e6fb627-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2e34517-339f-4a98-8892-abbc0e6fb627-maas-api-openshift-ca\") pod \"c2e34517-339f-4a98-8892-abbc0e6fb627\" (UID: \"c2e34517-339f-4a98-8892-abbc0e6fb627\") "
Oct 30 21:59:29.860373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:29.857056    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2e34517-339f-4a98-8892-abbc0e6fb627-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2e34517-339f-4a98-8892-abbc0e6fb627-maas-api-xks-ca\") pod \"c2e34517-339f-4a98-8892-abbc0e6fb627\" (UID: \"c2e34517-339f-4a98-8892-abbc0e6fb627\") "
Oct 30 21:59:29.860373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:29.857293    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2e34517-339f-4a98-8892-abbc0e6fb627-kube-api-access-djbgw\" (UniqueName: \"kubernetes.io/projected/c2e34517-339f-4a98-8892-abbc0e6fb627-kube-api-access-djbgw\") pod \"c2e34517-339f-4a98-8892-abbc0e6fb627\" (UID: \"c2e34517-339f-4a98-8892-abbc0e6fb627\") "
Oct 30 21:59:29.860373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:29.860096    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2e34517-339f-4a98-8892-abbc0e6fb627-maas-api-openshift-ca" pod "c2e34517-339f-4a98-8892-abbc0e6fb627" (UID: "c2e34517-339f-4a98-8892-abbc0e6fb627"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:59:29.870379 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:29.867546    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2e34517-339f-4a98-8892-abbc0e6fb627-kube-api-access-djbgw" pod "c2e34517-339f-4a98-8892-abbc0e6fb627" (UID: "c2e34517-339f-4a98-8892-abbc0e6fb627"). InnerVolumeSpecName "kube-api-access-djbgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:29.885553 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:29.880099    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2e34517-339f-4a98-8892-abbc0e6fb627-maas-api-xks-ca" pod "c2e34517-339f-4a98-8892-abbc0e6fb627" (UID: "c2e34517-339f-4a98-8892-abbc0e6fb627"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:59:29.934324 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:29.934243    5087 patch_prober.go:29] interesting pod/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused" start-of-body=
Oct 30 21:59:29.934324 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:29.934304    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd" podUID="3f71bde2-938d-4994-8a4e-6f37bbbee412" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.141:15021/healthz/ready\": dial tcp 10.132.0.141:15021: connect: connection refused"
Oct 30 21:59:29.974386 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:29.973740    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2e34517-339f-4a98-8892-abbc0e6fb627-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:59:29.974386 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:29.973778    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2e34517-339f-4a98-8892-abbc0e6fb627-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:59:29.974386 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:29.973793    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djbgw\" (UniqueName: \"kubernetes.io/projected/c2e34517-339f-4a98-8892-abbc0e6fb627-kube-api-access-djbgw\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:59:30.418367 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:30.418066    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f05b6443-2mkvj" event={"ID":"c2e34517-339f-4a98-8892-abbc0e6fb627","Type":"ContainerDied","Data":"4b2be7966580c4b324ec5d7b0f2a152fe89a5015b7632397f20c6522bc2c99df"}
Oct 30 21:59:30.418367 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:30.418135    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b2be7966580c4b324ec5d7b0f2a152fe89a5015b7632397f20c6522bc2c99df"
Oct 30 21:59:30.418367 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:30.418184    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f05b6443-2mkvj"
Oct 30 21:59:30.939827 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:30.939794    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:30.940497 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:30.940471    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:30.941289 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:30.941269    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 21:59:36.661588 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:36.661489    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f05b6443-fbbf556dc-bdtcf"]
Oct 30 21:59:40.061896 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:40.061857    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59cb4cce-x7z8c"]
Oct 30 21:59:40.110723 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:40.110688    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59cb4cce-x7z8c"]
Oct 30 21:59:41.038071 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:41.038039    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be51942b-p6jjv"]
Oct 30 21:59:41.038908 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:41.038884    5087 cpu_manager.go:414] "containerMap: removing container" podUID="c2e34517-339f-4a98-8892-abbc0e6fb627" containerName="revoke-keys"
Oct 30 21:59:41.038908 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:41.038907    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2e34517-339f-4a98-8892-abbc0e6fb627" containerName="revoke-keys"
Oct 30 21:59:41.039135 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:41.039120    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2e34517-339f-4a98-8892-abbc0e6fb627" containerName="revoke-keys"
Oct 30 21:59:41.042823 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:41.042800    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be51942b-p6jjv"
Oct 30 21:59:41.101941 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:41.101916    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be51942b-p6jjv"]
Oct 30 21:59:41.145554 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:41.145525    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:59:41.145713 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:41.145573    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:59:41.193781 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:41.193752    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-be51942b-p6jjv\" (UID: \"96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be51942b-p6jjv"
Oct 30 21:59:41.193913 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:41.193793    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8279t\" (UniqueName: \"kubernetes.io/projected/96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790-kube-api-access-8279t\") pod \"maas-api-revoke-keys-e2e-ait-be51942b-p6jjv\" (UID: \"96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be51942b-p6jjv"
Oct 30 21:59:41.194004 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:41.193985    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-be51942b-p6jjv\" (UID: \"96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be51942b-p6jjv"
Oct 30 21:59:41.295513 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:41.295426    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-be51942b-p6jjv\" (UID: \"96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be51942b-p6jjv"
Oct 30 21:59:41.295513 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:41.295467    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8279t\" (UniqueName: \"kubernetes.io/projected/96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790-kube-api-access-8279t\") pod \"maas-api-revoke-keys-e2e-ait-be51942b-p6jjv\" (UID: \"96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be51942b-p6jjv"
Oct 30 21:59:41.295711 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:41.295568    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-be51942b-p6jjv\" (UID: \"96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be51942b-p6jjv"
Oct 30 21:59:41.296192 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:41.296168    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-be51942b-p6jjv\" (UID: \"96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be51942b-p6jjv"
Oct 30 21:59:41.298089 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:41.298060    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-be51942b-p6jjv\" (UID: \"96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be51942b-p6jjv"
Oct 30 21:59:41.358234 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:41.358209    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8279t\" (UniqueName: \"kubernetes.io/projected/96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790-kube-api-access-8279t\") pod \"maas-api-revoke-keys-e2e-ait-be51942b-p6jjv\" (UID: \"96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be51942b-p6jjv"
Oct 30 21:59:41.657053 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:41.656981    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be51942b-p6jjv"
Oct 30 21:59:41.662145 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:41.662114    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-f05b6443-fbbf556dc-bdtcf" podUID="98f5afa5-3870-4279-b63f-d5971ab0c905" containerName="payload-processing" containerID="cri-o://3631a19f9c0aaac24db8f4581fddd7f843e28f79d9928ff90e1d2132a679db8e" gracePeriod=25
Oct 30 21:59:41.711902 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:41.711870    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7682ce92-4eb7-42b2-8d7d-8b898652367f" path="/var/lib/kubelet/pods/7682ce92-4eb7-42b2-8d7d-8b898652367f/volumes"
Oct 30 21:59:41.960550 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:41.960529    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f05b6443-fbbf556dc-bdtcf"
Oct 30 21:59:42.051337 ip-10-0-132-92 kubenswrapper[5087]: W1030 21:59:42.051305    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e6bfb5_5e6e_4c89_9d5b_e1bbf5e6c790.slice/crio-8be572b258faf9b2a834d95594c4a045a10e2b24248479239ec8b8ef306a7479 WatchSource:0}: Error finding container 8be572b258faf9b2a834d95594c4a045a10e2b24248479239ec8b8ef306a7479: Status 404 returned error can't find the container with id 8be572b258faf9b2a834d95594c4a045a10e2b24248479239ec8b8ef306a7479
Oct 30 21:59:42.052042 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:42.052019    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be51942b-p6jjv"]
Oct 30 21:59:42.107439 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:42.107414    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98f5afa5-3870-4279-b63f-d5971ab0c905-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98f5afa5-3870-4279-b63f-d5971ab0c905-plugins-config-volume\") pod \"98f5afa5-3870-4279-b63f-d5971ab0c905\" (UID: \"98f5afa5-3870-4279-b63f-d5971ab0c905\") "
Oct 30 21:59:42.107800 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:42.107632    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98f5afa5-3870-4279-b63f-d5971ab0c905-kube-api-access-n4n9z\" (UniqueName: \"kubernetes.io/projected/98f5afa5-3870-4279-b63f-d5971ab0c905-kube-api-access-n4n9z\") pod \"98f5afa5-3870-4279-b63f-d5971ab0c905\" (UID: \"98f5afa5-3870-4279-b63f-d5971ab0c905\") "
Oct 30 21:59:42.107849 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:42.107830    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98f5afa5-3870-4279-b63f-d5971ab0c905-plugins-config-volume" pod "98f5afa5-3870-4279-b63f-d5971ab0c905" (UID: "98f5afa5-3870-4279-b63f-d5971ab0c905"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:59:42.108444 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:42.108408    5087 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98f5afa5-3870-4279-b63f-d5971ab0c905-plugins-config-volume\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:59:42.110365 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:42.110327    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f5afa5-3870-4279-b63f-d5971ab0c905-kube-api-access-n4n9z" pod "98f5afa5-3870-4279-b63f-d5971ab0c905" (UID: "98f5afa5-3870-4279-b63f-d5971ab0c905"). InnerVolumeSpecName "kube-api-access-n4n9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:42.210244 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:42.210170    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4n9z\" (UniqueName: \"kubernetes.io/projected/98f5afa5-3870-4279-b63f-d5971ab0c905-kube-api-access-n4n9z\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:59:42.570401 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:42.570336    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be51942b-p6jjv" event={"ID":"96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790","Type":"ContainerStarted","Data":"49ccadc6c6235eb7f0a2269db72f4d77c190c5d942a668f9a2c800a19da3d7f7"}
Oct 30 21:59:42.570606 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:42.570448    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be51942b-p6jjv" event={"ID":"96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790","Type":"ContainerStarted","Data":"8be572b258faf9b2a834d95594c4a045a10e2b24248479239ec8b8ef306a7479"}
Oct 30 21:59:42.571931 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:42.571908    5087 generic.go:356] "Generic (PLEG): container finished" podID="98f5afa5-3870-4279-b63f-d5971ab0c905" containerID="3631a19f9c0aaac24db8f4581fddd7f843e28f79d9928ff90e1d2132a679db8e" exitCode=0
Oct 30 21:59:42.572056 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:42.571958    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f05b6443-fbbf556dc-bdtcf" event={"ID":"98f5afa5-3870-4279-b63f-d5971ab0c905","Type":"ContainerDied","Data":"3631a19f9c0aaac24db8f4581fddd7f843e28f79d9928ff90e1d2132a679db8e"}
Oct 30 21:59:42.572056 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:42.571985    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-f05b6443-fbbf556dc-bdtcf" event={"ID":"98f5afa5-3870-4279-b63f-d5971ab0c905","Type":"ContainerDied","Data":"b924d8f274cd14fd3bd091c3f43f5de74bdc21367a40cf4e2434af72152c56fc"}
Oct 30 21:59:42.572056 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:42.572004    5087 scope.go:122] "RemoveContainer" containerID="3631a19f9c0aaac24db8f4581fddd7f843e28f79d9928ff90e1d2132a679db8e"
Oct 30 21:59:42.572056 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:42.571957    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-f05b6443-fbbf556dc-bdtcf"
Oct 30 21:59:42.583871 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:42.583840    5087 scope.go:122] "RemoveContainer" containerID="3631a19f9c0aaac24db8f4581fddd7f843e28f79d9928ff90e1d2132a679db8e"
Oct 30 21:59:42.584224 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:59:42.584195    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3631a19f9c0aaac24db8f4581fddd7f843e28f79d9928ff90e1d2132a679db8e\": container with ID starting with 3631a19f9c0aaac24db8f4581fddd7f843e28f79d9928ff90e1d2132a679db8e not found: ID does not exist" containerID="3631a19f9c0aaac24db8f4581fddd7f843e28f79d9928ff90e1d2132a679db8e"
Oct 30 21:59:42.584377 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:42.584234    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3631a19f9c0aaac24db8f4581fddd7f843e28f79d9928ff90e1d2132a679db8e"} err="failed to get container status \"3631a19f9c0aaac24db8f4581fddd7f843e28f79d9928ff90e1d2132a679db8e\": rpc error: code = NotFound desc = could not find container \"3631a19f9c0aaac24db8f4581fddd7f843e28f79d9928ff90e1d2132a679db8e\": container with ID starting with 3631a19f9c0aaac24db8f4581fddd7f843e28f79d9928ff90e1d2132a679db8e not found: ID does not exist"
Oct 30 21:59:42.692174 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:42.692128    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be51942b-p6jjv" podStartSLOduration=2.692115974 podStartE2EDuration="2.692115974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:42.689134624 +0000 UTC m=+1577.163638604" watchObservedRunningTime="2026-10-30 21:59:42.692115974 +0000 UTC m=+1577.166619953"
Oct 30 21:59:42.817611 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:42.817563    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f05b6443-fbbf556dc-bdtcf"]
Oct 30 21:59:42.823099 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:42.823034    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-f05b6443-fbbf556dc-bdtcf"]
Oct 30 21:59:43.583532 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:43.583497    5087 generic.go:356] "Generic (PLEG): container finished" podID="96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790" containerID="49ccadc6c6235eb7f0a2269db72f4d77c190c5d942a668f9a2c800a19da3d7f7" exitCode=0
Oct 30 21:59:43.583954 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:43.583578    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be51942b-p6jjv" event={"ID":"96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790","Type":"ContainerDied","Data":"49ccadc6c6235eb7f0a2269db72f4d77c190c5d942a668f9a2c800a19da3d7f7"}
Oct 30 21:59:43.711558 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:43.711529    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98f5afa5-3870-4279-b63f-d5971ab0c905" path="/var/lib/kubelet/pods/98f5afa5-3870-4279-b63f-d5971ab0c905/volumes"
Oct 30 21:59:44.772133 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:44.772105    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be51942b-p6jjv"
Oct 30 21:59:44.950708 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:44.950618    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790-kube-api-access-8279t\" (UniqueName: \"kubernetes.io/projected/96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790-kube-api-access-8279t\") pod \"96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790\" (UID: \"96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790\") "
Oct 30 21:59:44.950922 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:44.950902    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790-maas-api-openshift-ca\") pod \"96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790\" (UID: \"96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790\") "
Oct 30 21:59:44.951046 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:44.951027    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790-maas-api-xks-ca\") pod \"96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790\" (UID: \"96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790\") "
Oct 30 21:59:44.951407 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:44.951310    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790-maas-api-openshift-ca" pod "96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790" (UID: "96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:59:44.951891 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:44.951813    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:59:44.953629 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:44.953572    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790-maas-api-xks-ca" pod "96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790" (UID: "96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:59:44.953629 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:44.953617    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790-kube-api-access-8279t" pod "96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790" (UID: "96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790"). InnerVolumeSpecName "kube-api-access-8279t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:45.053126 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:45.053099    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8279t\" (UniqueName: \"kubernetes.io/projected/96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790-kube-api-access-8279t\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:59:45.053126 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:45.053124    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:59:45.601733 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:45.601703    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be51942b-p6jjv"
Oct 30 21:59:45.601902 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:45.601698    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be51942b-p6jjv" event={"ID":"96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790","Type":"ContainerDied","Data":"8be572b258faf9b2a834d95594c4a045a10e2b24248479239ec8b8ef306a7479"}
Oct 30 21:59:45.601902 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:45.601842    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8be572b258faf9b2a834d95594c4a045a10e2b24248479239ec8b8ef306a7479"
Oct 30 21:59:47.172298 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:59:47.172262    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 21:59:48.903536 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:48.903499    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e611abd0-76s48"]
Oct 30 21:59:48.904431 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:48.904408    5087 cpu_manager.go:414] "containerMap: removing container" podUID="98f5afa5-3870-4279-b63f-d5971ab0c905" containerName="payload-processing"
Oct 30 21:59:48.904431 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:48.904430    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98f5afa5-3870-4279-b63f-d5971ab0c905" containerName="payload-processing"
Oct 30 21:59:48.904612 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:48.904486    5087 cpu_manager.go:414] "containerMap: removing container" podUID="96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790" containerName="revoke-keys"
Oct 30 21:59:48.904612 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:48.904494    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790" containerName="revoke-keys"
Oct 30 21:59:48.904721 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:48.904705    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790" containerName="revoke-keys"
Oct 30 21:59:48.904783 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:48.904735    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="98f5afa5-3870-4279-b63f-d5971ab0c905" containerName="payload-processing"
Oct 30 21:59:48.908573 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:48.908552    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e611abd0-76s48"
Oct 30 21:59:48.966424 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:48.966398    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e611abd0-76s48"]
Oct 30 21:59:49.009861 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:49.009829    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:59:49.010027 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:49.009892    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 21:59:49.010373 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:49.010322    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73395fa-cf8c-42d3-be5b-1f12e224d56f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e611abd0-76s48\" (UID: \"b73395fa-cf8c-42d3-be5b-1f12e224d56f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e611abd0-76s48"
Oct 30 21:59:49.010473 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:49.010423    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73395fa-cf8c-42d3-be5b-1f12e224d56f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e611abd0-76s48\" (UID: \"b73395fa-cf8c-42d3-be5b-1f12e224d56f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e611abd0-76s48"
Oct 30 21:59:49.010569 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:49.010525    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2nqx\" (UniqueName: \"kubernetes.io/projected/b73395fa-cf8c-42d3-be5b-1f12e224d56f-kube-api-access-h2nqx\") pod \"maas-api-revoke-sub-e611abd0-76s48\" (UID: \"b73395fa-cf8c-42d3-be5b-1f12e224d56f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e611abd0-76s48"
Oct 30 21:59:49.111790 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:49.111759    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73395fa-cf8c-42d3-be5b-1f12e224d56f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e611abd0-76s48\" (UID: \"b73395fa-cf8c-42d3-be5b-1f12e224d56f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e611abd0-76s48"
Oct 30 21:59:49.111951 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:49.111803    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73395fa-cf8c-42d3-be5b-1f12e224d56f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e611abd0-76s48\" (UID: \"b73395fa-cf8c-42d3-be5b-1f12e224d56f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e611abd0-76s48"
Oct 30 21:59:49.111951 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:49.111830    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2nqx\" (UniqueName: \"kubernetes.io/projected/b73395fa-cf8c-42d3-be5b-1f12e224d56f-kube-api-access-h2nqx\") pod \"maas-api-revoke-sub-e611abd0-76s48\" (UID: \"b73395fa-cf8c-42d3-be5b-1f12e224d56f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e611abd0-76s48"
Oct 30 21:59:49.112441 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:49.112420    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73395fa-cf8c-42d3-be5b-1f12e224d56f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e611abd0-76s48\" (UID: \"b73395fa-cf8c-42d3-be5b-1f12e224d56f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e611abd0-76s48"
Oct 30 21:59:49.114364 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:49.114317    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73395fa-cf8c-42d3-be5b-1f12e224d56f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e611abd0-76s48\" (UID: \"b73395fa-cf8c-42d3-be5b-1f12e224d56f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e611abd0-76s48"
Oct 30 21:59:49.168477 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:49.168413    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2nqx\" (UniqueName: \"kubernetes.io/projected/b73395fa-cf8c-42d3-be5b-1f12e224d56f-kube-api-access-h2nqx\") pod \"maas-api-revoke-sub-e611abd0-76s48\" (UID: \"b73395fa-cf8c-42d3-be5b-1f12e224d56f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e611abd0-76s48"
Oct 30 21:59:49.223469 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:49.223439    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e611abd0-76s48"
Oct 30 21:59:49.629566 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:49.629533    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e611abd0-76s48"]
Oct 30 21:59:50.649292 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:50.649254    5087 generic.go:356] "Generic (PLEG): container finished" podID="b73395fa-cf8c-42d3-be5b-1f12e224d56f" containerID="b86581bf3f90f338b9319b86f1a8244556673a4a21b513d3c0d73c4e0ec4e6c9" exitCode=0
Oct 30 21:59:50.649729 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:50.649401    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e611abd0-76s48" event={"ID":"b73395fa-cf8c-42d3-be5b-1f12e224d56f","Type":"ContainerDied","Data":"b86581bf3f90f338b9319b86f1a8244556673a4a21b513d3c0d73c4e0ec4e6c9"}
Oct 30 21:59:50.649729 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:50.649472    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e611abd0-76s48" event={"ID":"b73395fa-cf8c-42d3-be5b-1f12e224d56f","Type":"ContainerStarted","Data":"4745827bb62befadcef8b883ae64db57c44d4575eef53029e9f08b1b0f1ec146"}
Oct 30 21:59:51.854205 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:51.854179    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e611abd0-76s48"
Oct 30 21:59:51.962037 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:51.961964    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-be51942b-b8f988b87-bnklt"]
Oct 30 21:59:51.965814 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:51.965787    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b73395fa-cf8c-42d3-be5b-1f12e224d56f-kube-api-access-h2nqx\" (UniqueName: \"kubernetes.io/projected/b73395fa-cf8c-42d3-be5b-1f12e224d56f-kube-api-access-h2nqx\") pod \"b73395fa-cf8c-42d3-be5b-1f12e224d56f\" (UID: \"b73395fa-cf8c-42d3-be5b-1f12e224d56f\") "
Oct 30 21:59:51.965958 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:51.965907    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b73395fa-cf8c-42d3-be5b-1f12e224d56f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73395fa-cf8c-42d3-be5b-1f12e224d56f-maas-api-xks-ca\") pod \"b73395fa-cf8c-42d3-be5b-1f12e224d56f\" (UID: \"b73395fa-cf8c-42d3-be5b-1f12e224d56f\") "
Oct 30 21:59:51.965958 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:51.965943    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b73395fa-cf8c-42d3-be5b-1f12e224d56f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73395fa-cf8c-42d3-be5b-1f12e224d56f-maas-api-openshift-ca\") pod \"b73395fa-cf8c-42d3-be5b-1f12e224d56f\" (UID: \"b73395fa-cf8c-42d3-be5b-1f12e224d56f\") "
Oct 30 21:59:51.966412 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:51.966387    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b73395fa-cf8c-42d3-be5b-1f12e224d56f-maas-api-openshift-ca" pod "b73395fa-cf8c-42d3-be5b-1f12e224d56f" (UID: "b73395fa-cf8c-42d3-be5b-1f12e224d56f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:59:51.966714 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:51.966695    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b73395fa-cf8c-42d3-be5b-1f12e224d56f-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:59:51.968264 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:51.968233    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b73395fa-cf8c-42d3-be5b-1f12e224d56f-kube-api-access-h2nqx" pod "b73395fa-cf8c-42d3-be5b-1f12e224d56f" (UID: "b73395fa-cf8c-42d3-be5b-1f12e224d56f"). InnerVolumeSpecName "kube-api-access-h2nqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:51.968452 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:51.968400    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b73395fa-cf8c-42d3-be5b-1f12e224d56f-maas-api-xks-ca" pod "b73395fa-cf8c-42d3-be5b-1f12e224d56f" (UID: "b73395fa-cf8c-42d3-be5b-1f12e224d56f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:59:52.066574 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:52.066525    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a37b75cf-rbnsg"]
Oct 30 21:59:52.068420 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:52.068395    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b73395fa-cf8c-42d3-be5b-1f12e224d56f-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:59:52.068605 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:52.068580    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2nqx\" (UniqueName: \"kubernetes.io/projected/b73395fa-cf8c-42d3-be5b-1f12e224d56f-kube-api-access-h2nqx\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:59:52.074895 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:52.074873    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a37b75cf-rbnsg"]
Oct 30 21:59:52.667562 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:52.667526    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e611abd0-76s48"
Oct 30 21:59:52.667562 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:52.667535    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e611abd0-76s48" event={"ID":"b73395fa-cf8c-42d3-be5b-1f12e224d56f","Type":"ContainerDied","Data":"4745827bb62befadcef8b883ae64db57c44d4575eef53029e9f08b1b0f1ec146"}
Oct 30 21:59:52.667784 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:52.667603    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4745827bb62befadcef8b883ae64db57c44d4575eef53029e9f08b1b0f1ec146"
Oct 30 21:59:53.719835 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:53.719799    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d397505-2b12-4c67-8c4e-3e202bd77442" path="/var/lib/kubelet/pods/3d397505-2b12-4c67-8c4e-3e202bd77442/volumes"
Oct 30 21:59:56.963274 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:56.963218    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-be51942b-b8f988b87-bnklt" podUID="f157ab83-6db1-4d6a-b9f5-57ef3b0aa752" containerName="payload-pre-processing" containerID="cri-o://45d961f2b7d1ab8044f09b721031fec5c97dca70f2327897eebc6704f43d89b8" gracePeriod=25
Oct 30 21:59:57.309500 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:57.309476    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-be51942b-b8f988b87-bnklt"
Oct 30 21:59:57.367741 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:57.367705    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f157ab83-6db1-4d6a-b9f5-57ef3b0aa752-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f157ab83-6db1-4d6a-b9f5-57ef3b0aa752-plugins-config-volume\") pod \"f157ab83-6db1-4d6a-b9f5-57ef3b0aa752\" (UID: \"f157ab83-6db1-4d6a-b9f5-57ef3b0aa752\") "
Oct 30 21:59:57.367922 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:57.367838    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f157ab83-6db1-4d6a-b9f5-57ef3b0aa752-kube-api-access-zvqn9\" (UniqueName: \"kubernetes.io/projected/f157ab83-6db1-4d6a-b9f5-57ef3b0aa752-kube-api-access-zvqn9\") pod \"f157ab83-6db1-4d6a-b9f5-57ef3b0aa752\" (UID: \"f157ab83-6db1-4d6a-b9f5-57ef3b0aa752\") "
Oct 30 21:59:57.368141 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:57.368115    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f157ab83-6db1-4d6a-b9f5-57ef3b0aa752-plugins-config-volume" pod "f157ab83-6db1-4d6a-b9f5-57ef3b0aa752" (UID: "f157ab83-6db1-4d6a-b9f5-57ef3b0aa752"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:59:57.368584 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:57.368557    5087 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f157ab83-6db1-4d6a-b9f5-57ef3b0aa752-plugins-config-volume\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:59:57.370606 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:57.370579    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f157ab83-6db1-4d6a-b9f5-57ef3b0aa752-kube-api-access-zvqn9" pod "f157ab83-6db1-4d6a-b9f5-57ef3b0aa752" (UID: "f157ab83-6db1-4d6a-b9f5-57ef3b0aa752"). InnerVolumeSpecName "kube-api-access-zvqn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:57.469890 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:57.469849    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvqn9\" (UniqueName: \"kubernetes.io/projected/f157ab83-6db1-4d6a-b9f5-57ef3b0aa752-kube-api-access-zvqn9\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 21:59:57.716008 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:57.715913    5087 generic.go:356] "Generic (PLEG): container finished" podID="f157ab83-6db1-4d6a-b9f5-57ef3b0aa752" containerID="45d961f2b7d1ab8044f09b721031fec5c97dca70f2327897eebc6704f43d89b8" exitCode=0
Oct 30 21:59:57.716008 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:57.715979    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-be51942b-b8f988b87-bnklt"
Oct 30 21:59:57.716240 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:57.716010    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-be51942b-b8f988b87-bnklt" event={"ID":"f157ab83-6db1-4d6a-b9f5-57ef3b0aa752","Type":"ContainerDied","Data":"45d961f2b7d1ab8044f09b721031fec5c97dca70f2327897eebc6704f43d89b8"}
Oct 30 21:59:57.716240 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:57.716061    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-be51942b-b8f988b87-bnklt" event={"ID":"f157ab83-6db1-4d6a-b9f5-57ef3b0aa752","Type":"ContainerDied","Data":"62a99e8fe0b3a41391b6906f83c3580846e8799730d6f607d75214642eb4ce2e"}
Oct 30 21:59:57.716240 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:57.716082    5087 scope.go:122] "RemoveContainer" containerID="45d961f2b7d1ab8044f09b721031fec5c97dca70f2327897eebc6704f43d89b8"
Oct 30 21:59:57.735310 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:57.735283    5087 scope.go:122] "RemoveContainer" containerID="45d961f2b7d1ab8044f09b721031fec5c97dca70f2327897eebc6704f43d89b8"
Oct 30 21:59:57.735703 ip-10-0-132-92 kubenswrapper[5087]: E1030 21:59:57.735678    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45d961f2b7d1ab8044f09b721031fec5c97dca70f2327897eebc6704f43d89b8\": container with ID starting with 45d961f2b7d1ab8044f09b721031fec5c97dca70f2327897eebc6704f43d89b8 not found: ID does not exist" containerID="45d961f2b7d1ab8044f09b721031fec5c97dca70f2327897eebc6704f43d89b8"
Oct 30 21:59:57.735784 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:57.735713    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d961f2b7d1ab8044f09b721031fec5c97dca70f2327897eebc6704f43d89b8"} err="failed to get container status \"45d961f2b7d1ab8044f09b721031fec5c97dca70f2327897eebc6704f43d89b8\": rpc error: code = NotFound desc = could not find container \"45d961f2b7d1ab8044f09b721031fec5c97dca70f2327897eebc6704f43d89b8\": container with ID starting with 45d961f2b7d1ab8044f09b721031fec5c97dca70f2327897eebc6704f43d89b8 not found: ID does not exist"
Oct 30 21:59:57.889838 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:57.889806    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-be51942b-b8f988b87-bnklt"]
Oct 30 21:59:57.895700 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:57.895673    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-be51942b-b8f988b87-bnklt"]
Oct 30 21:59:59.711890 ip-10-0-132-92 kubenswrapper[5087]: I1030 21:59:59.711855    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f157ab83-6db1-4d6a-b9f5-57ef3b0aa752" path="/var/lib/kubelet/pods/f157ab83-6db1-4d6a-b9f5-57ef3b0aa752/volumes"
Oct 30 22:00:00.066136 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.066105    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg"]
Oct 30 22:00:00.066830 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.066814    5087 cpu_manager.go:414] "containerMap: removing container" podUID="f157ab83-6db1-4d6a-b9f5-57ef3b0aa752" containerName="payload-pre-processing"
Oct 30 22:00:00.066830 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.066831    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f157ab83-6db1-4d6a-b9f5-57ef3b0aa752" containerName="payload-pre-processing"
Oct 30 22:00:00.067008 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.066849    5087 cpu_manager.go:414] "containerMap: removing container" podUID="b73395fa-cf8c-42d3-be5b-1f12e224d56f" containerName="revoke-keys"
Oct 30 22:00:00.067008 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.066854    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b73395fa-cf8c-42d3-be5b-1f12e224d56f" containerName="revoke-keys"
Oct 30 22:00:00.067008 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.066970    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="b73395fa-cf8c-42d3-be5b-1f12e224d56f" containerName="revoke-keys"
Oct 30 22:00:00.067008 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.066987    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="f157ab83-6db1-4d6a-b9f5-57ef3b0aa752" containerName="payload-pre-processing"
Oct 30 22:00:00.070906 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.070876    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg"
Oct 30 22:00:00.124458 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.124430    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:00:00.130329 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.130310    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:00:00.179634 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.179599    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg"]
Oct 30 22:00:00.211881 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.211847    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6ac33a4-d857-4194-97ae-a43c681cbf43-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg\" (UID: \"c6ac33a4-d857-4194-97ae-a43c681cbf43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg"
Oct 30 22:00:00.212036 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.211897    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6ac33a4-d857-4194-97ae-a43c681cbf43-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg\" (UID: \"c6ac33a4-d857-4194-97ae-a43c681cbf43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg"
Oct 30 22:00:00.212036 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.211964    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcrms\" (UniqueName: \"kubernetes.io/projected/c6ac33a4-d857-4194-97ae-a43c681cbf43-kube-api-access-bcrms\") pod \"maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg\" (UID: \"c6ac33a4-d857-4194-97ae-a43c681cbf43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg"
Oct 30 22:00:00.215727 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.215704    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn"]
Oct 30 22:00:00.220439 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.220422    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn"
Oct 30 22:00:00.246492 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.246464    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk"]
Oct 30 22:00:00.252140 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.252116    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk"
Oct 30 22:00:00.254983 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.254960    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk"]
Oct 30 22:00:00.260498 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.259161    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk"
Oct 30 22:00:00.313995 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.313968    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcrms\" (UniqueName: \"kubernetes.io/projected/c6ac33a4-d857-4194-97ae-a43c681cbf43-kube-api-access-bcrms\") pod \"maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg\" (UID: \"c6ac33a4-d857-4194-97ae-a43c681cbf43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg"
Oct 30 22:00:00.314162 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.314040    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn\" (UID: \"c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn"
Oct 30 22:00:00.314271 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.314252    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdp9b\" (UniqueName: \"kubernetes.io/projected/c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd-kube-api-access-gdp9b\") pod \"maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn\" (UID: \"c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn"
Oct 30 22:00:00.314390 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.314375    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6ac33a4-d857-4194-97ae-a43c681cbf43-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg\" (UID: \"c6ac33a4-d857-4194-97ae-a43c681cbf43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg"
Oct 30 22:00:00.314458 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.314407    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6ac33a4-d857-4194-97ae-a43c681cbf43-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg\" (UID: \"c6ac33a4-d857-4194-97ae-a43c681cbf43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg"
Oct 30 22:00:00.315042 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.315021    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6ac33a4-d857-4194-97ae-a43c681cbf43-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg\" (UID: \"c6ac33a4-d857-4194-97ae-a43c681cbf43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg"
Oct 30 22:00:00.316771 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.316731    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6ac33a4-d857-4194-97ae-a43c681cbf43-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg\" (UID: \"c6ac33a4-d857-4194-97ae-a43c681cbf43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg"
Oct 30 22:00:00.333915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.333891    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn"]
Oct 30 22:00:00.363180 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.363146    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2"]
Oct 30 22:00:00.369446 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.369417    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2"
Oct 30 22:00:00.373259 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.373231    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj"]
Oct 30 22:00:00.378463 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.378439    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj"
Oct 30 22:00:00.381537 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.381514    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk"]
Oct 30 22:00:00.388486 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.388451    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8"]
Oct 30 22:00:00.389915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.389890    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcrms\" (UniqueName: \"kubernetes.io/projected/c6ac33a4-d857-4194-97ae-a43c681cbf43-kube-api-access-bcrms\") pod \"maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg\" (UID: \"c6ac33a4-d857-4194-97ae-a43c681cbf43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg"
Oct 30 22:00:00.392952 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.392934    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8"
Oct 30 22:00:00.416836 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.416810    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdp9b\" (UniqueName: \"kubernetes.io/projected/c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd-kube-api-access-gdp9b\") pod \"maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn\" (UID: \"c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn"
Oct 30 22:00:00.416947 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.416915    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8492282-c9d5-451f-b7ae-0a037a7e3442-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk\" (UID: \"e8492282-c9d5-451f-b7ae-0a037a7e3442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk"
Oct 30 22:00:00.417102 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.417080    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb9nl\" (UniqueName: \"kubernetes.io/projected/7981f8e3-b85e-4e26-8c57-21119cdd10fd-kube-api-access-mb9nl\") pod \"maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk\" (UID: \"7981f8e3-b85e-4e26-8c57-21119cdd10fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk"
Oct 30 22:00:00.417394 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.417370    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn\" (UID: \"c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn"
Oct 30 22:00:00.417514 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.417485    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2m8g\" (UniqueName: \"kubernetes.io/projected/e8492282-c9d5-451f-b7ae-0a037a7e3442-kube-api-access-q2m8g\") pod \"maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk\" (UID: \"e8492282-c9d5-451f-b7ae-0a037a7e3442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk"
Oct 30 22:00:00.417578 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.417555    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7981f8e3-b85e-4e26-8c57-21119cdd10fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk\" (UID: \"7981f8e3-b85e-4e26-8c57-21119cdd10fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk"
Oct 30 22:00:00.418384 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.418324    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn\" (UID: \"c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn"
Oct 30 22:00:00.442821 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.442792    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2"]
Oct 30 22:00:00.457205 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.457149    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj"]
Oct 30 22:00:00.466359 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.466313    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8"]
Oct 30 22:00:00.494434 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.494403    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdp9b\" (UniqueName: \"kubernetes.io/projected/c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd-kube-api-access-gdp9b\") pod \"maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn\" (UID: \"c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn"
Oct 30 22:00:00.520627 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.520593    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf7sh\" (UniqueName: \"kubernetes.io/projected/3176565c-81c7-4652-8b43-8f5ab5aa6f08-kube-api-access-qf7sh\") pod \"maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8\" (UID: \"3176565c-81c7-4652-8b43-8f5ab5aa6f08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8"
Oct 30 22:00:00.520783 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.520678    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3176565c-81c7-4652-8b43-8f5ab5aa6f08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8\" (UID: \"3176565c-81c7-4652-8b43-8f5ab5aa6f08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8"
Oct 30 22:00:00.520783 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.520754    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2m8g\" (UniqueName: \"kubernetes.io/projected/e8492282-c9d5-451f-b7ae-0a037a7e3442-kube-api-access-q2m8g\") pod \"maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk\" (UID: \"e8492282-c9d5-451f-b7ae-0a037a7e3442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk"
Oct 30 22:00:00.520903 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.520818    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7981f8e3-b85e-4e26-8c57-21119cdd10fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk\" (UID: \"7981f8e3-b85e-4e26-8c57-21119cdd10fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk"
Oct 30 22:00:00.520903 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.520882    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcgsw\" (UniqueName: \"kubernetes.io/projected/bccc6aee-c25a-4d9f-8d5c-32f417307675-kube-api-access-jcgsw\") pod \"maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2\" (UID: \"bccc6aee-c25a-4d9f-8d5c-32f417307675\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2"
Oct 30 22:00:00.521014 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.520980    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bccc6aee-c25a-4d9f-8d5c-32f417307675-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2\" (UID: \"bccc6aee-c25a-4d9f-8d5c-32f417307675\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2"
Oct 30 22:00:00.521067 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.521044    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8492282-c9d5-451f-b7ae-0a037a7e3442-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk\" (UID: \"e8492282-c9d5-451f-b7ae-0a037a7e3442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk"
Oct 30 22:00:00.521124 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.521090    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4688f9ff-10a5-45ff-9117-8a5bbd5adf37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj\" (UID: \"4688f9ff-10a5-45ff-9117-8a5bbd5adf37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj"
Oct 30 22:00:00.521175 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.521134    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6xtc\" (UniqueName: \"kubernetes.io/projected/4688f9ff-10a5-45ff-9117-8a5bbd5adf37-kube-api-access-n6xtc\") pod \"maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj\" (UID: \"4688f9ff-10a5-45ff-9117-8a5bbd5adf37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj"
Oct 30 22:00:00.521232 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.521198    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb9nl\" (UniqueName: \"kubernetes.io/projected/7981f8e3-b85e-4e26-8c57-21119cdd10fd-kube-api-access-mb9nl\") pod \"maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk\" (UID: \"7981f8e3-b85e-4e26-8c57-21119cdd10fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk"
Oct 30 22:00:00.521859 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.521824    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk"]
Oct 30 22:00:00.522455 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.522432    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7981f8e3-b85e-4e26-8c57-21119cdd10fd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk\" (UID: \"7981f8e3-b85e-4e26-8c57-21119cdd10fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk"
Oct 30 22:00:00.522792 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.522773    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8492282-c9d5-451f-b7ae-0a037a7e3442-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk\" (UID: \"e8492282-c9d5-451f-b7ae-0a037a7e3442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk"
Oct 30 22:00:00.534448 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.534429    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn"
Oct 30 22:00:00.591606 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.591526    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb9nl\" (UniqueName: \"kubernetes.io/projected/7981f8e3-b85e-4e26-8c57-21119cdd10fd-kube-api-access-mb9nl\") pod \"maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk\" (UID: \"7981f8e3-b85e-4e26-8c57-21119cdd10fd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk"
Oct 30 22:00:00.592554 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.592524    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2m8g\" (UniqueName: \"kubernetes.io/projected/e8492282-c9d5-451f-b7ae-0a037a7e3442-kube-api-access-q2m8g\") pod \"maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk\" (UID: \"e8492282-c9d5-451f-b7ae-0a037a7e3442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk"
Oct 30 22:00:00.595541 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.595516    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk"
Oct 30 22:00:00.624193 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.622496    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bccc6aee-c25a-4d9f-8d5c-32f417307675-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2\" (UID: \"bccc6aee-c25a-4d9f-8d5c-32f417307675\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2"
Oct 30 22:00:00.624193 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.622577    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4688f9ff-10a5-45ff-9117-8a5bbd5adf37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj\" (UID: \"4688f9ff-10a5-45ff-9117-8a5bbd5adf37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj"
Oct 30 22:00:00.624193 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.622620    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6xtc\" (UniqueName: \"kubernetes.io/projected/4688f9ff-10a5-45ff-9117-8a5bbd5adf37-kube-api-access-n6xtc\") pod \"maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj\" (UID: \"4688f9ff-10a5-45ff-9117-8a5bbd5adf37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj"
Oct 30 22:00:00.624193 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.623624    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bccc6aee-c25a-4d9f-8d5c-32f417307675-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2\" (UID: \"bccc6aee-c25a-4d9f-8d5c-32f417307675\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2"
Oct 30 22:00:00.628382 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.624880    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf7sh\" (UniqueName: \"kubernetes.io/projected/3176565c-81c7-4652-8b43-8f5ab5aa6f08-kube-api-access-qf7sh\") pod \"maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8\" (UID: \"3176565c-81c7-4652-8b43-8f5ab5aa6f08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8"
Oct 30 22:00:00.628382 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.624979    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3176565c-81c7-4652-8b43-8f5ab5aa6f08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8\" (UID: \"3176565c-81c7-4652-8b43-8f5ab5aa6f08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8"
Oct 30 22:00:00.628382 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.625148    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcgsw\" (UniqueName: \"kubernetes.io/projected/bccc6aee-c25a-4d9f-8d5c-32f417307675-kube-api-access-jcgsw\") pod \"maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2\" (UID: \"bccc6aee-c25a-4d9f-8d5c-32f417307675\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2"
Oct 30 22:00:00.628382 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.628266    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3176565c-81c7-4652-8b43-8f5ab5aa6f08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8\" (UID: \"3176565c-81c7-4652-8b43-8f5ab5aa6f08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8"
Oct 30 22:00:00.628382 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.628284    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4688f9ff-10a5-45ff-9117-8a5bbd5adf37-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj\" (UID: \"4688f9ff-10a5-45ff-9117-8a5bbd5adf37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj"
Oct 30 22:00:00.686306 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.686280    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg"
Oct 30 22:00:00.689725 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.689696    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6xtc\" (UniqueName: \"kubernetes.io/projected/4688f9ff-10a5-45ff-9117-8a5bbd5adf37-kube-api-access-n6xtc\") pod \"maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj\" (UID: \"4688f9ff-10a5-45ff-9117-8a5bbd5adf37\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj"
Oct 30 22:00:00.690062 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.690013    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf7sh\" (UniqueName: \"kubernetes.io/projected/3176565c-81c7-4652-8b43-8f5ab5aa6f08-kube-api-access-qf7sh\") pod \"maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8\" (UID: \"3176565c-81c7-4652-8b43-8f5ab5aa6f08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8"
Oct 30 22:00:00.691703 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.691247    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcgsw\" (UniqueName: \"kubernetes.io/projected/bccc6aee-c25a-4d9f-8d5c-32f417307675-kube-api-access-jcgsw\") pod \"maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2\" (UID: \"bccc6aee-c25a-4d9f-8d5c-32f417307675\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2"
Oct 30 22:00:00.714038 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.714012    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj"
Oct 30 22:00:00.723650 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.723624    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8"
Oct 30 22:00:00.877263 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.877083    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk"
Oct 30 22:00:00.955864 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:00:00.955824    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c0c03c_d543_4959_b0ac_6c87fb1ee8cd.slice/crio-d76050dd5ceef16d15a922ca00f7e4a554e93555c54e132f7cb33c54ade8c3d0 WatchSource:0}: Error finding container d76050dd5ceef16d15a922ca00f7e4a554e93555c54e132f7cb33c54ade8c3d0: Status 404 returned error can't find the container with id d76050dd5ceef16d15a922ca00f7e4a554e93555c54e132f7cb33c54ade8c3d0
Oct 30 22:00:00.988657 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.988462    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2"
Oct 30 22:00:00.991074 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:00.991045    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn"]
Oct 30 22:00:01.056269 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:01.054243    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk"]
Oct 30 22:00:01.126910 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:01.126722    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg"]
Oct 30 22:00:01.157642 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:00:01.157009    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ac33a4_d857_4194_97ae_a43c681cbf43.slice/crio-31e07612ac70b6a9a7e13d65cc02019bcd85c29911158d597ae9fe9b35806cfa WatchSource:0}: Error finding container 31e07612ac70b6a9a7e13d65cc02019bcd85c29911158d597ae9fe9b35806cfa: Status 404 returned error can't find the container with id 31e07612ac70b6a9a7e13d65cc02019bcd85c29911158d597ae9fe9b35806cfa
Oct 30 22:00:01.216072 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:00:01.216043    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4688f9ff_10a5_45ff_9117_8a5bbd5adf37.slice/crio-3949814a95f110e569700bdc6c0bc8d7e004520e511e890190659610cbd11d02 WatchSource:0}: Error finding container 3949814a95f110e569700bdc6c0bc8d7e004520e511e890190659610cbd11d02: Status 404 returned error can't find the container with id 3949814a95f110e569700bdc6c0bc8d7e004520e511e890190659610cbd11d02
Oct 30 22:00:01.216180 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:01.216091    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8"]
Oct 30 22:00:01.230583 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:01.230551    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj"]
Oct 30 22:00:01.302455 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:01.302429    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk"]
Oct 30 22:00:01.349328 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:00:01.349298    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7981f8e3_b85e_4e26_8c57_21119cdd10fd.slice/crio-86697699f53087e26bd5c9eebb95560e836f54c3205ef0dea098ec231152947a WatchSource:0}: Error finding container 86697699f53087e26bd5c9eebb95560e836f54c3205ef0dea098ec231152947a: Status 404 returned error can't find the container with id 86697699f53087e26bd5c9eebb95560e836f54c3205ef0dea098ec231152947a
Oct 30 22:00:01.541477 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:01.538211    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2"]
Oct 30 22:00:01.764933 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:01.764894    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj" event={"ID":"4688f9ff-10a5-45ff-9117-8a5bbd5adf37","Type":"ContainerStarted","Data":"3949814a95f110e569700bdc6c0bc8d7e004520e511e890190659610cbd11d02"}
Oct 30 22:00:01.766406 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:01.766373    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk" event={"ID":"e8492282-c9d5-451f-b7ae-0a037a7e3442","Type":"ContainerStarted","Data":"1e58b3456d4cc356c5420b9985e226579ac9335b9c5fc172e238aef79880e0f8"}
Oct 30 22:00:01.767894 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:01.767866    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2" event={"ID":"bccc6aee-c25a-4d9f-8d5c-32f417307675","Type":"ContainerStarted","Data":"8d8c34e60bb6c009e16040ef74e09d72d143fb809106e9957cc3bcbf79b94e61"}
Oct 30 22:00:01.769670 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:01.769645    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn" event={"ID":"c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd","Type":"ContainerStarted","Data":"da9cdebbf4f9372d5537ede60cce31a4410f5279aaece1fe9edf273b9568bf03"}
Oct 30 22:00:01.769795 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:01.769692    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn" event={"ID":"c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd","Type":"ContainerStarted","Data":"d76050dd5ceef16d15a922ca00f7e4a554e93555c54e132f7cb33c54ade8c3d0"}
Oct 30 22:00:01.771554 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:01.771527    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg" event={"ID":"c6ac33a4-d857-4194-97ae-a43c681cbf43","Type":"ContainerStarted","Data":"d454de01d64bbb57a0c0f6dafb88a1f65af1e9f2f8ff750b8e0df2c0c3a9aa7e"}
Oct 30 22:00:01.771662 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:01.771568    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg" event={"ID":"c6ac33a4-d857-4194-97ae-a43c681cbf43","Type":"ContainerStarted","Data":"31e07612ac70b6a9a7e13d65cc02019bcd85c29911158d597ae9fe9b35806cfa"}
Oct 30 22:00:01.772871 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:01.772848    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk" event={"ID":"7981f8e3-b85e-4e26-8c57-21119cdd10fd","Type":"ContainerStarted","Data":"86697699f53087e26bd5c9eebb95560e836f54c3205ef0dea098ec231152947a"}
Oct 30 22:00:01.774180 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:01.774162    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8" event={"ID":"3176565c-81c7-4652-8b43-8f5ab5aa6f08","Type":"ContainerStarted","Data":"53b9ffd5574438dbf6c3931ea9524ea45757c566696ba6c3068fe136519e889c"}
Oct 30 22:00:01.885410 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:01.885227    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn" podStartSLOduration=1.885212168 podStartE2EDuration="1.885212168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:01.884544757 +0000 UTC m=+1596.359048730" watchObservedRunningTime="2026-10-30 22:00:01.885212168 +0000 UTC m=+1596.359716149"
Oct 30 22:00:02.014360 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:02.014297    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg" podStartSLOduration=2.014281726 podStartE2EDuration="2.014281726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:02.008792363 +0000 UTC m=+1596.483296341" watchObservedRunningTime="2026-10-30 22:00:02.014281726 +0000 UTC m=+1596.488785706"
Oct 30 22:00:02.787339 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:02.787311    5087 generic.go:356] "Generic (PLEG): container finished" podID="3176565c-81c7-4652-8b43-8f5ab5aa6f08" containerID="c3c80070435c09af677b12b7183e694b7bc91f93a0dd321398e4befc51cfd460" exitCode=60
Oct 30 22:00:02.787810 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:02.787396    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8" event={"ID":"3176565c-81c7-4652-8b43-8f5ab5aa6f08","Type":"ContainerDied","Data":"c3c80070435c09af677b12b7183e694b7bc91f93a0dd321398e4befc51cfd460"}
Oct 30 22:00:02.787810 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:02.787708    5087 scope.go:122] "RemoveContainer" containerID="c3c80070435c09af677b12b7183e694b7bc91f93a0dd321398e4befc51cfd460"
Oct 30 22:00:02.789579 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:02.789555    5087 generic.go:356] "Generic (PLEG): container finished" podID="4688f9ff-10a5-45ff-9117-8a5bbd5adf37" containerID="e47ca9079f468c4bddb7d5b60d37ea7c8c89fdadc7982f1dc195be4a67776719" exitCode=60
Oct 30 22:00:02.789703 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:02.789626    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj" event={"ID":"4688f9ff-10a5-45ff-9117-8a5bbd5adf37","Type":"ContainerDied","Data":"e47ca9079f468c4bddb7d5b60d37ea7c8c89fdadc7982f1dc195be4a67776719"}
Oct 30 22:00:02.789881 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:02.789865    5087 scope.go:122] "RemoveContainer" containerID="e47ca9079f468c4bddb7d5b60d37ea7c8c89fdadc7982f1dc195be4a67776719"
Oct 30 22:00:02.791423 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:02.791253    5087 generic.go:356] "Generic (PLEG): container finished" podID="e8492282-c9d5-451f-b7ae-0a037a7e3442" containerID="d320adb4d89c878d34cfab5451e50ab3930e907f519757a6952c626ca6193ea7" exitCode=60
Oct 30 22:00:02.791423 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:02.791278    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk" event={"ID":"e8492282-c9d5-451f-b7ae-0a037a7e3442","Type":"ContainerDied","Data":"d320adb4d89c878d34cfab5451e50ab3930e907f519757a6952c626ca6193ea7"}
Oct 30 22:00:02.791613 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:02.791594    5087 scope.go:122] "RemoveContainer" containerID="d320adb4d89c878d34cfab5451e50ab3930e907f519757a6952c626ca6193ea7"
Oct 30 22:00:02.793198 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:02.793112    5087 generic.go:356] "Generic (PLEG): container finished" podID="bccc6aee-c25a-4d9f-8d5c-32f417307675" containerID="579df4dd87515619f7373d22395cf85af7d799d0b56ec2cc60ba55b2c1448152" exitCode=60
Oct 30 22:00:02.793198 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:02.793133    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2" event={"ID":"bccc6aee-c25a-4d9f-8d5c-32f417307675","Type":"ContainerDied","Data":"579df4dd87515619f7373d22395cf85af7d799d0b56ec2cc60ba55b2c1448152"}
Oct 30 22:00:02.793455 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:02.793438    5087 scope.go:122] "RemoveContainer" containerID="579df4dd87515619f7373d22395cf85af7d799d0b56ec2cc60ba55b2c1448152"
Oct 30 22:00:02.795058 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:02.795035    5087 generic.go:356] "Generic (PLEG): container finished" podID="c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd" containerID="da9cdebbf4f9372d5537ede60cce31a4410f5279aaece1fe9edf273b9568bf03" exitCode=60
Oct 30 22:00:02.795166 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:02.795068    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn" event={"ID":"c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd","Type":"ContainerDied","Data":"da9cdebbf4f9372d5537ede60cce31a4410f5279aaece1fe9edf273b9568bf03"}
Oct 30 22:00:02.795576 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:02.795556    5087 scope.go:122] "RemoveContainer" containerID="da9cdebbf4f9372d5537ede60cce31a4410f5279aaece1fe9edf273b9568bf03"
Oct 30 22:00:02.797247 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:02.797229    5087 generic.go:356] "Generic (PLEG): container finished" podID="c6ac33a4-d857-4194-97ae-a43c681cbf43" containerID="d454de01d64bbb57a0c0f6dafb88a1f65af1e9f2f8ff750b8e0df2c0c3a9aa7e" exitCode=0
Oct 30 22:00:02.797406 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:02.797311    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg" event={"ID":"c6ac33a4-d857-4194-97ae-a43c681cbf43","Type":"ContainerDied","Data":"d454de01d64bbb57a0c0f6dafb88a1f65af1e9f2f8ff750b8e0df2c0c3a9aa7e"}
Oct 30 22:00:02.799057 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:02.799039    5087 generic.go:356] "Generic (PLEG): container finished" podID="7981f8e3-b85e-4e26-8c57-21119cdd10fd" containerID="a3a0813b41f40685b5a499247405cf69628c68b0887f8f93959afc2c0886ac96" exitCode=60
Oct 30 22:00:02.799136 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:02.799097    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk" event={"ID":"7981f8e3-b85e-4e26-8c57-21119cdd10fd","Type":"ContainerDied","Data":"a3a0813b41f40685b5a499247405cf69628c68b0887f8f93959afc2c0886ac96"}
Oct 30 22:00:02.799304 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:02.799292    5087 scope.go:122] "RemoveContainer" containerID="a3a0813b41f40685b5a499247405cf69628c68b0887f8f93959afc2c0886ac96"
Oct 30 22:00:03.813909 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:03.813873    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk" event={"ID":"7981f8e3-b85e-4e26-8c57-21119cdd10fd","Type":"ContainerStarted","Data":"44fbe5289dc0032374c9bf842da966a7a4eb938afae01c1cdbff06a8bc9f6662"}
Oct 30 22:00:03.816000 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:03.815974    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8" event={"ID":"3176565c-81c7-4652-8b43-8f5ab5aa6f08","Type":"ContainerStarted","Data":"988b37b9f677adb645ffc13777b71b1af9e29ecded7afbcb34f97ec1bc26fb4f"}
Oct 30 22:00:03.817893 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:03.817869    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj" event={"ID":"4688f9ff-10a5-45ff-9117-8a5bbd5adf37","Type":"ContainerStarted","Data":"f2cbf9adeed30ceb2bc27e52e0b985ce6459300d8ea9833578f4bfe6204a2ec5"}
Oct 30 22:00:03.819689 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:03.819658    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk" event={"ID":"e8492282-c9d5-451f-b7ae-0a037a7e3442","Type":"ContainerStarted","Data":"84df03173b1f39ff07b7a7819d4dd34e85cb423d826d100c0583fd02731a7286"}
Oct 30 22:00:03.821501 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:03.821475    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2" event={"ID":"bccc6aee-c25a-4d9f-8d5c-32f417307675","Type":"ContainerStarted","Data":"59042aae9439c3172962881908ee0fbc214606098382c4b09b6ff9a1edaf424a"}
Oct 30 22:00:03.833636 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:03.833609    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn" event={"ID":"c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd","Type":"ContainerStarted","Data":"e685599cef6eac370b3042ce2c5c0ab50cbd8bbb388481b5eeed23d2c5138120"}
Oct 30 22:00:03.934285 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:03.934230    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk" podStartSLOduration=3.934215856 podStartE2EDuration="3.934215856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:03.930915028 +0000 UTC m=+1598.405419008" watchObservedRunningTime="2026-10-30 22:00:03.934215856 +0000 UTC m=+1598.408719835"
Oct 30 22:00:04.055787 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.055730    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8" podStartSLOduration=4.05571057 podStartE2EDuration="4.05571057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:04.04803666 +0000 UTC m=+1598.522540650" watchObservedRunningTime="2026-10-30 22:00:04.05571057 +0000 UTC m=+1598.530214550"
Oct 30 22:00:04.169165 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.169109    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2" podStartSLOduration=4.169091936 podStartE2EDuration="4.169091936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:04.165065789 +0000 UTC m=+1598.639569793" watchObservedRunningTime="2026-10-30 22:00:04.169091936 +0000 UTC m=+1598.643595916"
Oct 30 22:00:04.251229 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.251206    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg"
Oct 30 22:00:04.308929 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.308901    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6ac33a4-d857-4194-97ae-a43c681cbf43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6ac33a4-d857-4194-97ae-a43c681cbf43-maas-api-xks-ca\") pod \"c6ac33a4-d857-4194-97ae-a43c681cbf43\" (UID: \"c6ac33a4-d857-4194-97ae-a43c681cbf43\") "
Oct 30 22:00:04.309070 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.308962    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6ac33a4-d857-4194-97ae-a43c681cbf43-kube-api-access-bcrms\" (UniqueName: \"kubernetes.io/projected/c6ac33a4-d857-4194-97ae-a43c681cbf43-kube-api-access-bcrms\") pod \"c6ac33a4-d857-4194-97ae-a43c681cbf43\" (UID: \"c6ac33a4-d857-4194-97ae-a43c681cbf43\") "
Oct 30 22:00:04.309269 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.309251    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6ac33a4-d857-4194-97ae-a43c681cbf43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6ac33a4-d857-4194-97ae-a43c681cbf43-maas-api-openshift-ca\") pod \"c6ac33a4-d857-4194-97ae-a43c681cbf43\" (UID: \"c6ac33a4-d857-4194-97ae-a43c681cbf43\") "
Oct 30 22:00:04.309626 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.309603    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6ac33a4-d857-4194-97ae-a43c681cbf43-maas-api-openshift-ca" pod "c6ac33a4-d857-4194-97ae-a43c681cbf43" (UID: "c6ac33a4-d857-4194-97ae-a43c681cbf43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:04.309958 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.309941    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6ac33a4-d857-4194-97ae-a43c681cbf43-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:04.311112 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.311090    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6ac33a4-d857-4194-97ae-a43c681cbf43-maas-api-xks-ca" pod "c6ac33a4-d857-4194-97ae-a43c681cbf43" (UID: "c6ac33a4-d857-4194-97ae-a43c681cbf43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:00:04.311187 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.311112    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6ac33a4-d857-4194-97ae-a43c681cbf43-kube-api-access-bcrms" pod "c6ac33a4-d857-4194-97ae-a43c681cbf43" (UID: "c6ac33a4-d857-4194-97ae-a43c681cbf43"). InnerVolumeSpecName "kube-api-access-bcrms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:04.406549 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.406506    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk" podStartSLOduration=4.406493491 podStartE2EDuration="4.406493491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:04.405604911 +0000 UTC m=+1598.880108891" watchObservedRunningTime="2026-10-30 22:00:04.406493491 +0000 UTC m=+1598.880997464"
Oct 30 22:00:04.409808 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.409770    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj" podStartSLOduration=4.409758883 podStartE2EDuration="4.409758883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:04.283789778 +0000 UTC m=+1598.758293757" watchObservedRunningTime="2026-10-30 22:00:04.409758883 +0000 UTC m=+1598.884262862"
Oct 30 22:00:04.411478 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.411447    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6ac33a4-d857-4194-97ae-a43c681cbf43-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:04.411596 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.411484    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcrms\" (UniqueName: \"kubernetes.io/projected/c6ac33a4-d857-4194-97ae-a43c681cbf43-kube-api-access-bcrms\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:04.847033 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.847001    5087 generic.go:356] "Generic (PLEG): container finished" podID="bccc6aee-c25a-4d9f-8d5c-32f417307675" containerID="59042aae9439c3172962881908ee0fbc214606098382c4b09b6ff9a1edaf424a" exitCode=60
Oct 30 22:00:04.847553 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.847093    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2" event={"ID":"bccc6aee-c25a-4d9f-8d5c-32f417307675","Type":"ContainerDied","Data":"59042aae9439c3172962881908ee0fbc214606098382c4b09b6ff9a1edaf424a"}
Oct 30 22:00:04.847553 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.847165    5087 scope.go:122] "RemoveContainer" containerID="579df4dd87515619f7373d22395cf85af7d799d0b56ec2cc60ba55b2c1448152"
Oct 30 22:00:04.847553 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.847403    5087 scope.go:122] "RemoveContainer" containerID="59042aae9439c3172962881908ee0fbc214606098382c4b09b6ff9a1edaf424a"
Oct 30 22:00:04.847802 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:00:04.847777    5087 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2_odh-ai-gateway-infra(bccc6aee-c25a-4d9f-8d5c-32f417307675)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2" podUID="bccc6aee-c25a-4d9f-8d5c-32f417307675"
Oct 30 22:00:04.849731 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.849708    5087 generic.go:356] "Generic (PLEG): container finished" podID="c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd" containerID="e685599cef6eac370b3042ce2c5c0ab50cbd8bbb388481b5eeed23d2c5138120" exitCode=60
Oct 30 22:00:04.849836 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.849781    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn" event={"ID":"c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd","Type":"ContainerDied","Data":"e685599cef6eac370b3042ce2c5c0ab50cbd8bbb388481b5eeed23d2c5138120"}
Oct 30 22:00:04.850037 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.850020    5087 scope.go:122] "RemoveContainer" containerID="e685599cef6eac370b3042ce2c5c0ab50cbd8bbb388481b5eeed23d2c5138120"
Oct 30 22:00:04.850489 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:00:04.850378    5087 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn_odh-ai-gateway-infra(c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn" podUID="c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd"
Oct 30 22:00:04.851533 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.851513    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg"
Oct 30 22:00:04.851622 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.851544    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg" event={"ID":"c6ac33a4-d857-4194-97ae-a43c681cbf43","Type":"ContainerDied","Data":"31e07612ac70b6a9a7e13d65cc02019bcd85c29911158d597ae9fe9b35806cfa"}
Oct 30 22:00:04.851622 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.851587    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31e07612ac70b6a9a7e13d65cc02019bcd85c29911158d597ae9fe9b35806cfa"
Oct 30 22:00:04.853811 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.853789    5087 generic.go:356] "Generic (PLEG): container finished" podID="7981f8e3-b85e-4e26-8c57-21119cdd10fd" containerID="44fbe5289dc0032374c9bf842da966a7a4eb938afae01c1cdbff06a8bc9f6662" exitCode=60
Oct 30 22:00:04.853909 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.853897    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk" event={"ID":"7981f8e3-b85e-4e26-8c57-21119cdd10fd","Type":"ContainerDied","Data":"44fbe5289dc0032374c9bf842da966a7a4eb938afae01c1cdbff06a8bc9f6662"}
Oct 30 22:00:04.854298 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.854283    5087 scope.go:122] "RemoveContainer" containerID="44fbe5289dc0032374c9bf842da966a7a4eb938afae01c1cdbff06a8bc9f6662"
Oct 30 22:00:04.854718 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:00:04.854694    5087 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk_odh-ai-gateway-infra(7981f8e3-b85e-4e26-8c57-21119cdd10fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk" podUID="7981f8e3-b85e-4e26-8c57-21119cdd10fd"
Oct 30 22:00:04.856326 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.856308    5087 generic.go:356] "Generic (PLEG): container finished" podID="3176565c-81c7-4652-8b43-8f5ab5aa6f08" containerID="988b37b9f677adb645ffc13777b71b1af9e29ecded7afbcb34f97ec1bc26fb4f" exitCode=60
Oct 30 22:00:04.856558 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.856540    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8" event={"ID":"3176565c-81c7-4652-8b43-8f5ab5aa6f08","Type":"ContainerDied","Data":"988b37b9f677adb645ffc13777b71b1af9e29ecded7afbcb34f97ec1bc26fb4f"}
Oct 30 22:00:04.856910 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.856880    5087 scope.go:122] "RemoveContainer" containerID="988b37b9f677adb645ffc13777b71b1af9e29ecded7afbcb34f97ec1bc26fb4f"
Oct 30 22:00:04.857182 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:00:04.857160    5087 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8_odh-ai-gateway-infra(3176565c-81c7-4652-8b43-8f5ab5aa6f08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8" podUID="3176565c-81c7-4652-8b43-8f5ab5aa6f08"
Oct 30 22:00:04.859294 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.859272    5087 generic.go:356] "Generic (PLEG): container finished" podID="4688f9ff-10a5-45ff-9117-8a5bbd5adf37" containerID="f2cbf9adeed30ceb2bc27e52e0b985ce6459300d8ea9833578f4bfe6204a2ec5" exitCode=60
Oct 30 22:00:04.859414 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.859381    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj" event={"ID":"4688f9ff-10a5-45ff-9117-8a5bbd5adf37","Type":"ContainerDied","Data":"f2cbf9adeed30ceb2bc27e52e0b985ce6459300d8ea9833578f4bfe6204a2ec5"}
Oct 30 22:00:04.859574 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.859557    5087 scope.go:122] "RemoveContainer" containerID="f2cbf9adeed30ceb2bc27e52e0b985ce6459300d8ea9833578f4bfe6204a2ec5"
Oct 30 22:00:04.859797 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:00:04.859771    5087 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj_odh-ai-gateway-infra(4688f9ff-10a5-45ff-9117-8a5bbd5adf37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj" podUID="4688f9ff-10a5-45ff-9117-8a5bbd5adf37"
Oct 30 22:00:04.862148 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.862125    5087 generic.go:356] "Generic (PLEG): container finished" podID="e8492282-c9d5-451f-b7ae-0a037a7e3442" containerID="84df03173b1f39ff07b7a7819d4dd34e85cb423d826d100c0583fd02731a7286" exitCode=60
Oct 30 22:00:04.862230 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.862170    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk" event={"ID":"e8492282-c9d5-451f-b7ae-0a037a7e3442","Type":"ContainerDied","Data":"84df03173b1f39ff07b7a7819d4dd34e85cb423d826d100c0583fd02731a7286"}
Oct 30 22:00:04.862735 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.862524    5087 scope.go:122] "RemoveContainer" containerID="84df03173b1f39ff07b7a7819d4dd34e85cb423d826d100c0583fd02731a7286"
Oct 30 22:00:04.862820 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:00:04.862797    5087 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk_odh-ai-gateway-infra(e8492282-c9d5-451f-b7ae-0a037a7e3442)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk" podUID="e8492282-c9d5-451f-b7ae-0a037a7e3442"
Oct 30 22:00:04.863981 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.863959    5087 scope.go:122] "RemoveContainer" containerID="da9cdebbf4f9372d5537ede60cce31a4410f5279aaece1fe9edf273b9568bf03"
Oct 30 22:00:04.880231 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.879697    5087 scope.go:122] "RemoveContainer" containerID="a3a0813b41f40685b5a499247405cf69628c68b0887f8f93959afc2c0886ac96"
Oct 30 22:00:04.893942 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.893910    5087 scope.go:122] "RemoveContainer" containerID="c3c80070435c09af677b12b7183e694b7bc91f93a0dd321398e4befc51cfd460"
Oct 30 22:00:04.907473 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.907456    5087 scope.go:122] "RemoveContainer" containerID="e47ca9079f468c4bddb7d5b60d37ea7c8c89fdadc7982f1dc195be4a67776719"
Oct 30 22:00:04.921892 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:04.921862    5087 scope.go:122] "RemoveContainer" containerID="d320adb4d89c878d34cfab5451e50ab3930e907f519757a6952c626ca6193ea7"
Oct 30 22:00:05.876640 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:05.876602    5087 scope.go:122] "RemoveContainer" containerID="44fbe5289dc0032374c9bf842da966a7a4eb938afae01c1cdbff06a8bc9f6662"
Oct 30 22:00:05.877016 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:00:05.876989    5087 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk_odh-ai-gateway-infra(7981f8e3-b85e-4e26-8c57-21119cdd10fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk" podUID="7981f8e3-b85e-4e26-8c57-21119cdd10fd"
Oct 30 22:00:05.880091 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:05.880059    5087 scope.go:122] "RemoveContainer" containerID="988b37b9f677adb645ffc13777b71b1af9e29ecded7afbcb34f97ec1bc26fb4f"
Oct 30 22:00:05.880561 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:00:05.880521    5087 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8_odh-ai-gateway-infra(3176565c-81c7-4652-8b43-8f5ab5aa6f08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8" podUID="3176565c-81c7-4652-8b43-8f5ab5aa6f08"
Oct 30 22:00:05.882210 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:05.882176    5087 scope.go:122] "RemoveContainer" containerID="f2cbf9adeed30ceb2bc27e52e0b985ce6459300d8ea9833578f4bfe6204a2ec5"
Oct 30 22:00:05.882650 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:00:05.882580    5087 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj_odh-ai-gateway-infra(4688f9ff-10a5-45ff-9117-8a5bbd5adf37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj" podUID="4688f9ff-10a5-45ff-9117-8a5bbd5adf37"
Oct 30 22:00:05.884459 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:05.884436    5087 scope.go:122] "RemoveContainer" containerID="84df03173b1f39ff07b7a7819d4dd34e85cb423d826d100c0583fd02731a7286"
Oct 30 22:00:05.884823 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:00:05.884797    5087 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk_odh-ai-gateway-infra(e8492282-c9d5-451f-b7ae-0a037a7e3442)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk" podUID="e8492282-c9d5-451f-b7ae-0a037a7e3442"
Oct 30 22:00:05.886545 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:05.886515    5087 scope.go:122] "RemoveContainer" containerID="59042aae9439c3172962881908ee0fbc214606098382c4b09b6ff9a1edaf424a"
Oct 30 22:00:05.886857 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:00:05.886830    5087 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2_odh-ai-gateway-infra(bccc6aee-c25a-4d9f-8d5c-32f417307675)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2" podUID="bccc6aee-c25a-4d9f-8d5c-32f417307675"
Oct 30 22:00:05.888325 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:05.888306    5087 scope.go:122] "RemoveContainer" containerID="e685599cef6eac370b3042ce2c5c0ab50cbd8bbb388481b5eeed23d2c5138120"
Oct 30 22:00:05.888662 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:00:05.888643    5087 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn_odh-ai-gateway-infra(c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn" podUID="c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd"
Oct 30 22:00:05.968808 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:05.968769    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8"]
Oct 30 22:00:07.251542 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:07.251512    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8"
Oct 30 22:00:07.390306 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:07.390271    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3176565c-81c7-4652-8b43-8f5ab5aa6f08-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3176565c-81c7-4652-8b43-8f5ab5aa6f08-maas-api-service-ca\") pod \"3176565c-81c7-4652-8b43-8f5ab5aa6f08\" (UID: \"3176565c-81c7-4652-8b43-8f5ab5aa6f08\") "
Oct 30 22:00:07.390541 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:07.390315    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3176565c-81c7-4652-8b43-8f5ab5aa6f08-kube-api-access-qf7sh\" (UniqueName: \"kubernetes.io/projected/3176565c-81c7-4652-8b43-8f5ab5aa6f08-kube-api-access-qf7sh\") pod \"3176565c-81c7-4652-8b43-8f5ab5aa6f08\" (UID: \"3176565c-81c7-4652-8b43-8f5ab5aa6f08\") "
Oct 30 22:00:07.390718 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:07.390694    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3176565c-81c7-4652-8b43-8f5ab5aa6f08-maas-api-service-ca" pod "3176565c-81c7-4652-8b43-8f5ab5aa6f08" (UID: "3176565c-81c7-4652-8b43-8f5ab5aa6f08"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:07.391073 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:07.391012    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3176565c-81c7-4652-8b43-8f5ab5aa6f08-maas-api-service-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:07.396939 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:07.396909    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3176565c-81c7-4652-8b43-8f5ab5aa6f08-kube-api-access-qf7sh" pod "3176565c-81c7-4652-8b43-8f5ab5aa6f08" (UID: "3176565c-81c7-4652-8b43-8f5ab5aa6f08"). InnerVolumeSpecName "kube-api-access-qf7sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:07.492242 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:07.492208    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf7sh\" (UniqueName: \"kubernetes.io/projected/3176565c-81c7-4652-8b43-8f5ab5aa6f08-kube-api-access-qf7sh\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:07.905995 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:07.905914    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8"
Oct 30 22:00:07.905995 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:07.905952    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8" event={"ID":"3176565c-81c7-4652-8b43-8f5ab5aa6f08","Type":"ContainerDied","Data":"53b9ffd5574438dbf6c3931ea9524ea45757c566696ba6c3068fe136519e889c"}
Oct 30 22:00:07.906235 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:07.906024    5087 scope.go:122] "RemoveContainer" containerID="988b37b9f677adb645ffc13777b71b1af9e29ecded7afbcb34f97ec1bc26fb4f"
Oct 30 22:00:08.123041 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:08.123013    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8"]
Oct 30 22:00:08.128120 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:08.128098    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-f05b6443-29889960-h9kx8"]
Oct 30 22:00:08.821285 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:08.821246    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f05b6443-2mkvj"]
Oct 30 22:00:08.875400 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:08.875338    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f05b6443-2mkvj"]
Oct 30 22:00:09.712019 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:09.711980    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3176565c-81c7-4652-8b43-8f5ab5aa6f08" path="/var/lib/kubelet/pods/3176565c-81c7-4652-8b43-8f5ab5aa6f08/volumes"
Oct 30 22:00:09.712769 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:09.712749    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2e34517-339f-4a98-8892-abbc0e6fb627" path="/var/lib/kubelet/pods/c2e34517-339f-4a98-8892-abbc0e6fb627/volumes"
Oct 30 22:00:14.077372 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:14.068242    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84075163-469ss"]
Oct 30 22:00:14.127400 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:14.125648    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84075163-469ss"]
Oct 30 22:00:14.737188 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:14.734835    5087 scope.go:122] "RemoveContainer" containerID="44fbe5289dc0032374c9bf842da966a7a4eb938afae01c1cdbff06a8bc9f6662"
Oct 30 22:00:15.707834 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:15.707801    5087 scope.go:122] "RemoveContainer" containerID="f2cbf9adeed30ceb2bc27e52e0b985ce6459300d8ea9833578f4bfe6204a2ec5"
Oct 30 22:00:15.712930 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:15.712909    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bc41ae4-0b46-4e51-83ec-c9224f91072d" path="/var/lib/kubelet/pods/1bc41ae4-0b46-4e51-83ec-c9224f91072d/volumes"
Oct 30 22:00:15.999583 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:15.999552    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be51942b-p6jjv"]
Oct 30 22:00:16.016655 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:16.016626    5087 generic.go:356] "Generic (PLEG): container finished" podID="7981f8e3-b85e-4e26-8c57-21119cdd10fd" containerID="dc1816212a6a6ceae24dfd4170190b7eaf7799eb55aabf6c3e871681900e8ac3" exitCode=60
Oct 30 22:00:16.016798 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:16.016701    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk" event={"ID":"7981f8e3-b85e-4e26-8c57-21119cdd10fd","Type":"ContainerDied","Data":"dc1816212a6a6ceae24dfd4170190b7eaf7799eb55aabf6c3e871681900e8ac3"}
Oct 30 22:00:16.016798 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:16.016759    5087 scope.go:122] "RemoveContainer" containerID="44fbe5289dc0032374c9bf842da966a7a4eb938afae01c1cdbff06a8bc9f6662"
Oct 30 22:00:16.017153 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:16.017137    5087 scope.go:122] "RemoveContainer" containerID="dc1816212a6a6ceae24dfd4170190b7eaf7799eb55aabf6c3e871681900e8ac3"
Oct 30 22:00:16.017577 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:00:16.017547    5087 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk_odh-ai-gateway-infra(7981f8e3-b85e-4e26-8c57-21119cdd10fd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk" podUID="7981f8e3-b85e-4e26-8c57-21119cdd10fd"
Oct 30 22:00:16.048482 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:16.048456    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-be51942b-p6jjv"]
Oct 30 22:00:16.710679 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:16.704245    5087 scope.go:122] "RemoveContainer" containerID="e685599cef6eac370b3042ce2c5c0ab50cbd8bbb388481b5eeed23d2c5138120"
Oct 30 22:00:17.054369 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:17.050885    5087 generic.go:356] "Generic (PLEG): container finished" podID="4688f9ff-10a5-45ff-9117-8a5bbd5adf37" containerID="15d62bfedb542de01056b2bf5bc5dab748d6961136135f70c97e6e95dc318da0" exitCode=60
Oct 30 22:00:17.054369 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:17.051088    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj" event={"ID":"4688f9ff-10a5-45ff-9117-8a5bbd5adf37","Type":"ContainerDied","Data":"15d62bfedb542de01056b2bf5bc5dab748d6961136135f70c97e6e95dc318da0"}
Oct 30 22:00:17.054369 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:17.051150    5087 scope.go:122] "RemoveContainer" containerID="f2cbf9adeed30ceb2bc27e52e0b985ce6459300d8ea9833578f4bfe6204a2ec5"
Oct 30 22:00:17.054369 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:17.051822    5087 scope.go:122] "RemoveContainer" containerID="15d62bfedb542de01056b2bf5bc5dab748d6961136135f70c97e6e95dc318da0"
Oct 30 22:00:17.054369 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:00:17.052397    5087 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj_odh-ai-gateway-infra(4688f9ff-10a5-45ff-9117-8a5bbd5adf37)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj" podUID="4688f9ff-10a5-45ff-9117-8a5bbd5adf37"
Oct 30 22:00:17.169832 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:17.169792    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk"]
Oct 30 22:00:17.268576 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:00:17.268526    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:00:17.456226 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:17.451271    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk"
Oct 30 22:00:17.527900 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:17.527867    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7981f8e3-b85e-4e26-8c57-21119cdd10fd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7981f8e3-b85e-4e26-8c57-21119cdd10fd-maas-api-service-ca\") pod \"7981f8e3-b85e-4e26-8c57-21119cdd10fd\" (UID: \"7981f8e3-b85e-4e26-8c57-21119cdd10fd\") "
Oct 30 22:00:17.528233 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:17.528211    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7981f8e3-b85e-4e26-8c57-21119cdd10fd-kube-api-access-mb9nl\" (UniqueName: \"kubernetes.io/projected/7981f8e3-b85e-4e26-8c57-21119cdd10fd-kube-api-access-mb9nl\") pod \"7981f8e3-b85e-4e26-8c57-21119cdd10fd\" (UID: \"7981f8e3-b85e-4e26-8c57-21119cdd10fd\") "
Oct 30 22:00:17.528766 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:17.528718    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7981f8e3-b85e-4e26-8c57-21119cdd10fd-maas-api-service-ca" pod "7981f8e3-b85e-4e26-8c57-21119cdd10fd" (UID: "7981f8e3-b85e-4e26-8c57-21119cdd10fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:17.528973 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:17.528948    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7981f8e3-b85e-4e26-8c57-21119cdd10fd-maas-api-service-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:17.531271 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:17.531250    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7981f8e3-b85e-4e26-8c57-21119cdd10fd-kube-api-access-mb9nl" pod "7981f8e3-b85e-4e26-8c57-21119cdd10fd" (UID: "7981f8e3-b85e-4e26-8c57-21119cdd10fd"). InnerVolumeSpecName "kube-api-access-mb9nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:17.630101 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:17.630070    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb9nl\" (UniqueName: \"kubernetes.io/projected/7981f8e3-b85e-4e26-8c57-21119cdd10fd-kube-api-access-mb9nl\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:17.712682 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:17.712606    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790" path="/var/lib/kubelet/pods/96e6bfb5-5e6e-4c89-9d5b-e1bbf5e6c790/volumes"
Oct 30 22:00:18.079381 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:18.074406    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-497e3152-26zm9"]
Oct 30 22:00:18.086232 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:18.080613    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-497e3152-26zm9"]
Oct 30 22:00:18.086232 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:18.085579    5087 generic.go:356] "Generic (PLEG): container finished" podID="c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd" containerID="dc85bed1550ab678f55b3884ed761013a521155b67f8111585113767d9271fd6" exitCode=60
Oct 30 22:00:18.086232 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:18.085709    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn" event={"ID":"c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd","Type":"ContainerDied","Data":"dc85bed1550ab678f55b3884ed761013a521155b67f8111585113767d9271fd6"}
Oct 30 22:00:18.086232 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:18.085762    5087 scope.go:122] "RemoveContainer" containerID="e685599cef6eac370b3042ce2c5c0ab50cbd8bbb388481b5eeed23d2c5138120"
Oct 30 22:00:18.086580 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:18.086297    5087 scope.go:122] "RemoveContainer" containerID="dc85bed1550ab678f55b3884ed761013a521155b67f8111585113767d9271fd6"
Oct 30 22:00:18.091017 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:00:18.090968    5087 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn_odh-ai-gateway-infra(c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn" podUID="c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd"
Oct 30 22:00:18.094097 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:18.094069    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk" event={"ID":"7981f8e3-b85e-4e26-8c57-21119cdd10fd","Type":"ContainerDied","Data":"86697699f53087e26bd5c9eebb95560e836f54c3205ef0dea098ec231152947a"}
Oct 30 22:00:18.094220 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:18.094173    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk"
Oct 30 22:00:18.137398 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:18.135879    5087 scope.go:122] "RemoveContainer" containerID="dc1816212a6a6ceae24dfd4170190b7eaf7799eb55aabf6c3e871681900e8ac3"
Oct 30 22:00:18.206771 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:18.206732    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj"]
Oct 30 22:00:18.331499 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:18.329263    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk"]
Oct 30 22:00:18.337531 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:18.337491    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-e585c53f-29889960-hwwzk"]
Oct 30 22:00:18.404281 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:18.404259    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj"
Oct 30 22:00:18.549147 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:18.549111    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4688f9ff-10a5-45ff-9117-8a5bbd5adf37-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4688f9ff-10a5-45ff-9117-8a5bbd5adf37-maas-api-service-ca\") pod \"4688f9ff-10a5-45ff-9117-8a5bbd5adf37\" (UID: \"4688f9ff-10a5-45ff-9117-8a5bbd5adf37\") "
Oct 30 22:00:18.549326 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:18.549254    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4688f9ff-10a5-45ff-9117-8a5bbd5adf37-kube-api-access-n6xtc\" (UniqueName: \"kubernetes.io/projected/4688f9ff-10a5-45ff-9117-8a5bbd5adf37-kube-api-access-n6xtc\") pod \"4688f9ff-10a5-45ff-9117-8a5bbd5adf37\" (UID: \"4688f9ff-10a5-45ff-9117-8a5bbd5adf37\") "
Oct 30 22:00:18.549486 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:18.549463    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4688f9ff-10a5-45ff-9117-8a5bbd5adf37-maas-api-service-ca" pod "4688f9ff-10a5-45ff-9117-8a5bbd5adf37" (UID: "4688f9ff-10a5-45ff-9117-8a5bbd5adf37"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:18.550155 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:18.550132    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4688f9ff-10a5-45ff-9117-8a5bbd5adf37-maas-api-service-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:18.551747 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:18.551724    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4688f9ff-10a5-45ff-9117-8a5bbd5adf37-kube-api-access-n6xtc" pod "4688f9ff-10a5-45ff-9117-8a5bbd5adf37" (UID: "4688f9ff-10a5-45ff-9117-8a5bbd5adf37"). InnerVolumeSpecName "kube-api-access-n6xtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:18.652194 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:18.652114    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6xtc\" (UniqueName: \"kubernetes.io/projected/4688f9ff-10a5-45ff-9117-8a5bbd5adf37-kube-api-access-n6xtc\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:18.702829 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:18.702798    5087 scope.go:122] "RemoveContainer" containerID="59042aae9439c3172962881908ee0fbc214606098382c4b09b6ff9a1edaf424a"
Oct 30 22:00:18.702996 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:18.702944    5087 scope.go:122] "RemoveContainer" containerID="84df03173b1f39ff07b7a7819d4dd34e85cb423d826d100c0583fd02731a7286"
Oct 30 22:00:19.131822 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:19.131004    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj" event={"ID":"4688f9ff-10a5-45ff-9117-8a5bbd5adf37","Type":"ContainerDied","Data":"3949814a95f110e569700bdc6c0bc8d7e004520e511e890190659610cbd11d02"}
Oct 30 22:00:19.131822 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:19.131102    5087 scope.go:122] "RemoveContainer" containerID="15d62bfedb542de01056b2bf5bc5dab748d6961136135f70c97e6e95dc318da0"
Oct 30 22:00:19.131822 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:19.131217    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj"
Oct 30 22:00:19.233695 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:19.233659    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn"]
Oct 30 22:00:19.320459 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:19.320390    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj"]
Oct 30 22:00:19.333002 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:19.332965    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-557708-29889960-sfkqj"]
Oct 30 22:00:19.436097 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:19.436073    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn"
Oct 30 22:00:19.586751 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:19.586720    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd-kube-api-access-gdp9b\" (UniqueName: \"kubernetes.io/projected/c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd-kube-api-access-gdp9b\") pod \"c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd\" (UID: \"c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd\") "
Oct 30 22:00:19.586922 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:19.586873    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd-maas-api-service-ca\") pod \"c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd\" (UID: \"c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd\") "
Oct 30 22:00:19.587240 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:19.587216    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd-maas-api-service-ca" pod "c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd" (UID: "c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:19.587703 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:19.587685    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd-maas-api-service-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:19.589060 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:19.589035    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd-kube-api-access-gdp9b" pod "c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd" (UID: "c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd"). InnerVolumeSpecName "kube-api-access-gdp9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:19.689403 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:19.689305    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdp9b\" (UniqueName: \"kubernetes.io/projected/c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd-kube-api-access-gdp9b\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:19.715276 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:19.715241    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4688f9ff-10a5-45ff-9117-8a5bbd5adf37" path="/var/lib/kubelet/pods/4688f9ff-10a5-45ff-9117-8a5bbd5adf37/volumes"
Oct 30 22:00:19.715978 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:19.715953    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6514e520-3f58-46dd-9158-46cb05110a94" path="/var/lib/kubelet/pods/6514e520-3f58-46dd-9158-46cb05110a94/volumes"
Oct 30 22:00:19.716523 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:19.716504    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7981f8e3-b85e-4e26-8c57-21119cdd10fd" path="/var/lib/kubelet/pods/7981f8e3-b85e-4e26-8c57-21119cdd10fd/volumes"
Oct 30 22:00:20.143893 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:20.143863    5087 generic.go:356] "Generic (PLEG): container finished" podID="e8492282-c9d5-451f-b7ae-0a037a7e3442" containerID="baa68f41efaa315ba06f1a570be6e110134b52d7ecc5d40805c177316c9074bf" exitCode=60
Oct 30 22:00:20.144415 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:20.143965    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk" event={"ID":"e8492282-c9d5-451f-b7ae-0a037a7e3442","Type":"ContainerDied","Data":"baa68f41efaa315ba06f1a570be6e110134b52d7ecc5d40805c177316c9074bf"}
Oct 30 22:00:20.144415 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:20.144034    5087 scope.go:122] "RemoveContainer" containerID="84df03173b1f39ff07b7a7819d4dd34e85cb423d826d100c0583fd02731a7286"
Oct 30 22:00:20.144892 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:20.144616    5087 scope.go:122] "RemoveContainer" containerID="baa68f41efaa315ba06f1a570be6e110134b52d7ecc5d40805c177316c9074bf"
Oct 30 22:00:20.145125 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:00:20.145080    5087 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk_odh-ai-gateway-infra(e8492282-c9d5-451f-b7ae-0a037a7e3442)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk" podUID="e8492282-c9d5-451f-b7ae-0a037a7e3442"
Oct 30 22:00:20.147840 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:20.147821    5087 generic.go:356] "Generic (PLEG): container finished" podID="bccc6aee-c25a-4d9f-8d5c-32f417307675" containerID="23e7520e8c9273ba98d56f669c0cce32332565d2f867a57057c6622e9d4364b8" exitCode=60
Oct 30 22:00:20.147922 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:20.147900    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2" event={"ID":"bccc6aee-c25a-4d9f-8d5c-32f417307675","Type":"ContainerDied","Data":"23e7520e8c9273ba98d56f669c0cce32332565d2f867a57057c6622e9d4364b8"}
Oct 30 22:00:20.148341 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:20.148322    5087 scope.go:122] "RemoveContainer" containerID="23e7520e8c9273ba98d56f669c0cce32332565d2f867a57057c6622e9d4364b8"
Oct 30 22:00:20.148712 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:00:20.148688    5087 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2_odh-ai-gateway-infra(bccc6aee-c25a-4d9f-8d5c-32f417307675)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2" podUID="bccc6aee-c25a-4d9f-8d5c-32f417307675"
Oct 30 22:00:20.149453 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:20.149335    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn"
Oct 30 22:00:20.149568 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:20.149411    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn" event={"ID":"c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd","Type":"ContainerDied","Data":"d76050dd5ceef16d15a922ca00f7e4a554e93555c54e132f7cb33c54ade8c3d0"}
Oct 30 22:00:20.159721 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:20.159705    5087 scope.go:122] "RemoveContainer" containerID="59042aae9439c3172962881908ee0fbc214606098382c4b09b6ff9a1edaf424a"
Oct 30 22:00:20.174085 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:20.174063    5087 scope.go:122] "RemoveContainer" containerID="dc85bed1550ab678f55b3884ed761013a521155b67f8111585113767d9271fd6"
Oct 30 22:00:20.396106 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:20.396067    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn"]
Oct 30 22:00:20.401057 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:20.401028    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c79cb5-29889960-65kzn"]
Oct 30 22:00:21.317381 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:21.317330    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk"]
Oct 30 22:00:21.595825 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:21.595750    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2"]
Oct 30 22:00:21.635223 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:21.635193    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk"
Oct 30 22:00:21.717737 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:21.717366    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd" path="/var/lib/kubelet/pods/c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd/volumes"
Oct 30 22:00:21.765693 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:21.765644    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8492282-c9d5-451f-b7ae-0a037a7e3442-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8492282-c9d5-451f-b7ae-0a037a7e3442-maas-api-service-ca\") pod \"e8492282-c9d5-451f-b7ae-0a037a7e3442\" (UID: \"e8492282-c9d5-451f-b7ae-0a037a7e3442\") "
Oct 30 22:00:21.765836 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:21.765731    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8492282-c9d5-451f-b7ae-0a037a7e3442-kube-api-access-q2m8g\" (UniqueName: \"kubernetes.io/projected/e8492282-c9d5-451f-b7ae-0a037a7e3442-kube-api-access-q2m8g\") pod \"e8492282-c9d5-451f-b7ae-0a037a7e3442\" (UID: \"e8492282-c9d5-451f-b7ae-0a037a7e3442\") "
Oct 30 22:00:21.767539 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:21.767505    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8492282-c9d5-451f-b7ae-0a037a7e3442-maas-api-service-ca" pod "e8492282-c9d5-451f-b7ae-0a037a7e3442" (UID: "e8492282-c9d5-451f-b7ae-0a037a7e3442"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:21.770684 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:21.770657    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8492282-c9d5-451f-b7ae-0a037a7e3442-kube-api-access-q2m8g" pod "e8492282-c9d5-451f-b7ae-0a037a7e3442" (UID: "e8492282-c9d5-451f-b7ae-0a037a7e3442"). InnerVolumeSpecName "kube-api-access-q2m8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:21.832053 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:21.832022    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2"
Oct 30 22:00:21.872782 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:21.872371    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bccc6aee-c25a-4d9f-8d5c-32f417307675-kube-api-access-jcgsw\" (UniqueName: \"kubernetes.io/projected/bccc6aee-c25a-4d9f-8d5c-32f417307675-kube-api-access-jcgsw\") pod \"bccc6aee-c25a-4d9f-8d5c-32f417307675\" (UID: \"bccc6aee-c25a-4d9f-8d5c-32f417307675\") "
Oct 30 22:00:21.872782 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:21.872543    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bccc6aee-c25a-4d9f-8d5c-32f417307675-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bccc6aee-c25a-4d9f-8d5c-32f417307675-maas-api-service-ca\") pod \"bccc6aee-c25a-4d9f-8d5c-32f417307675\" (UID: \"bccc6aee-c25a-4d9f-8d5c-32f417307675\") "
Oct 30 22:00:21.873457 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:21.873364    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e8492282-c9d5-451f-b7ae-0a037a7e3442-maas-api-service-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:21.873457 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:21.873396    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2m8g\" (UniqueName: \"kubernetes.io/projected/e8492282-c9d5-451f-b7ae-0a037a7e3442-kube-api-access-q2m8g\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:21.873974 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:21.873942    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bccc6aee-c25a-4d9f-8d5c-32f417307675-maas-api-service-ca" pod "bccc6aee-c25a-4d9f-8d5c-32f417307675" (UID: "bccc6aee-c25a-4d9f-8d5c-32f417307675"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:21.906074 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:21.906036    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bccc6aee-c25a-4d9f-8d5c-32f417307675-kube-api-access-jcgsw" pod "bccc6aee-c25a-4d9f-8d5c-32f417307675" (UID: "bccc6aee-c25a-4d9f-8d5c-32f417307675"). InnerVolumeSpecName "kube-api-access-jcgsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:21.979651 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:21.979506    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcgsw\" (UniqueName: \"kubernetes.io/projected/bccc6aee-c25a-4d9f-8d5c-32f417307675-kube-api-access-jcgsw\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:21.979968 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:21.979927    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bccc6aee-c25a-4d9f-8d5c-32f417307675-maas-api-service-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:22.188470 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:22.188380    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk" event={"ID":"e8492282-c9d5-451f-b7ae-0a037a7e3442","Type":"ContainerDied","Data":"1e58b3456d4cc356c5420b9985e226579ac9335b9c5fc172e238aef79880e0f8"}
Oct 30 22:00:22.188709 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:22.188691    5087 scope.go:122] "RemoveContainer" containerID="baa68f41efaa315ba06f1a570be6e110134b52d7ecc5d40805c177316c9074bf"
Oct 30 22:00:22.188816 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:22.188423    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk"
Oct 30 22:00:22.190955 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:22.190927    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2"
Oct 30 22:00:22.191166 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:22.191085    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2" event={"ID":"bccc6aee-c25a-4d9f-8d5c-32f417307675","Type":"ContainerDied","Data":"8d8c34e60bb6c009e16040ef74e09d72d143fb809106e9957cc3bcbf79b94e61"}
Oct 30 22:00:22.204958 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:22.204893    5087 scope.go:122] "RemoveContainer" containerID="23e7520e8c9273ba98d56f669c0cce32332565d2f867a57057c6622e9d4364b8"
Oct 30 22:00:22.361563 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:22.361523    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2"]
Oct 30 22:00:22.366275 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:22.366247    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889960-g49s2"]
Oct 30 22:00:22.470588 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:22.470505    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk"]
Oct 30 22:00:22.478310 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:22.478280    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-80c366-29889960-jzqgk"]
Oct 30 22:00:23.710736 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:23.710708    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bccc6aee-c25a-4d9f-8d5c-32f417307675" path="/var/lib/kubelet/pods/bccc6aee-c25a-4d9f-8d5c-32f417307675/volumes"
Oct 30 22:00:23.711240 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:23.711124    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8492282-c9d5-451f-b7ae-0a037a7e3442" path="/var/lib/kubelet/pods/e8492282-c9d5-451f-b7ae-0a037a7e3442/volumes"
Oct 30 22:00:27.737380 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:27.734054    5087 scope.go:122] "RemoveContainer" containerID="fb1bfb3ffb0ca132e02cec33dec62beb79875a620a46a8d55eaf80d567ce0b4a"
Oct 30 22:00:28.066139 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:28.066058    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba05e518-z46fv"]
Oct 30 22:00:28.113821 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:28.113786    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ba05e518-z46fv"]
Oct 30 22:00:29.269401 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:29.269357    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg"]
Oct 30 22:00:29.317884 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:29.317857    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e585c53f-4zmhg"]
Oct 30 22:00:29.797152 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:29.797120    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73e4cb78-cbd4-43aa-a9c7-5418323158db" path="/var/lib/kubelet/pods/73e4cb78-cbd4-43aa-a9c7-5418323158db/volumes"
Oct 30 22:00:29.797901 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:29.797875    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6ac33a4-d857-4194-97ae-a43c681cbf43" path="/var/lib/kubelet/pods/c6ac33a4-d857-4194-97ae-a43c681cbf43/volumes"
Oct 30 22:00:30.751100 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:30.751064    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"]
Oct 30 22:00:30.751573 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:30.751311    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd" podUID="3f71bde2-938d-4994-8a4e-6f37bbbee412" containerName="istio-proxy" containerID="cri-o://8f86807b09a71d7b481cb4f47603445bf891eab4df9022dd164870c32144998f" gracePeriod=30
Oct 30 22:00:30.973370 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:30.969378    5087 patch_prober.go:29] interesting pod/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:00:30.973370 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:30.969453    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd" podUID="3f71bde2-938d-4994-8a4e-6f37bbbee412" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:00:34.054092 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.054038    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caaa84af-7s79k"]
Oct 30 22:00:34.055103 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055081    5087 cpu_manager.go:414] "containerMap: removing container" podUID="bccc6aee-c25a-4d9f-8d5c-32f417307675" containerName="cleanup"
Oct 30 22:00:34.055214 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055107    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bccc6aee-c25a-4d9f-8d5c-32f417307675" containerName="cleanup"
Oct 30 22:00:34.055214 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055127    5087 cpu_manager.go:414] "containerMap: removing container" podUID="4688f9ff-10a5-45ff-9117-8a5bbd5adf37" containerName="cleanup"
Oct 30 22:00:34.055214 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055134    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4688f9ff-10a5-45ff-9117-8a5bbd5adf37" containerName="cleanup"
Oct 30 22:00:34.055214 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055144    5087 cpu_manager.go:414] "containerMap: removing container" podUID="4688f9ff-10a5-45ff-9117-8a5bbd5adf37" containerName="cleanup"
Oct 30 22:00:34.055214 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055152    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4688f9ff-10a5-45ff-9117-8a5bbd5adf37" containerName="cleanup"
Oct 30 22:00:34.055214 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055167    5087 cpu_manager.go:414] "containerMap: removing container" podUID="4688f9ff-10a5-45ff-9117-8a5bbd5adf37" containerName="cleanup"
Oct 30 22:00:34.055214 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055174    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4688f9ff-10a5-45ff-9117-8a5bbd5adf37" containerName="cleanup"
Oct 30 22:00:34.055214 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055212    5087 cpu_manager.go:414] "containerMap: removing container" podUID="c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd" containerName="cleanup"
Oct 30 22:00:34.055681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055219    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd" containerName="cleanup"
Oct 30 22:00:34.055681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055233    5087 cpu_manager.go:414] "containerMap: removing container" podUID="bccc6aee-c25a-4d9f-8d5c-32f417307675" containerName="cleanup"
Oct 30 22:00:34.055681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055240    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bccc6aee-c25a-4d9f-8d5c-32f417307675" containerName="cleanup"
Oct 30 22:00:34.055681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055262    5087 cpu_manager.go:414] "containerMap: removing container" podUID="e8492282-c9d5-451f-b7ae-0a037a7e3442" containerName="cleanup"
Oct 30 22:00:34.055681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055269    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8492282-c9d5-451f-b7ae-0a037a7e3442" containerName="cleanup"
Oct 30 22:00:34.055681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055284    5087 cpu_manager.go:414] "containerMap: removing container" podUID="3176565c-81c7-4652-8b43-8f5ab5aa6f08" containerName="cleanup"
Oct 30 22:00:34.055681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055291    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3176565c-81c7-4652-8b43-8f5ab5aa6f08" containerName="cleanup"
Oct 30 22:00:34.055681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055305    5087 cpu_manager.go:414] "containerMap: removing container" podUID="7981f8e3-b85e-4e26-8c57-21119cdd10fd" containerName="cleanup"
Oct 30 22:00:34.055681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055313    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7981f8e3-b85e-4e26-8c57-21119cdd10fd" containerName="cleanup"
Oct 30 22:00:34.055681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055324    5087 cpu_manager.go:414] "containerMap: removing container" podUID="c6ac33a4-d857-4194-97ae-a43c681cbf43" containerName="revoke-keys"
Oct 30 22:00:34.055681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055330    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6ac33a4-d857-4194-97ae-a43c681cbf43" containerName="revoke-keys"
Oct 30 22:00:34.055681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055339    5087 cpu_manager.go:414] "containerMap: removing container" podUID="e8492282-c9d5-451f-b7ae-0a037a7e3442" containerName="cleanup"
Oct 30 22:00:34.055681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055361    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8492282-c9d5-451f-b7ae-0a037a7e3442" containerName="cleanup"
Oct 30 22:00:34.055681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055384    5087 cpu_manager.go:414] "containerMap: removing container" podUID="3176565c-81c7-4652-8b43-8f5ab5aa6f08" containerName="cleanup"
Oct 30 22:00:34.055681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055391    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3176565c-81c7-4652-8b43-8f5ab5aa6f08" containerName="cleanup"
Oct 30 22:00:34.055681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055403    5087 cpu_manager.go:414] "containerMap: removing container" podUID="c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd" containerName="cleanup"
Oct 30 22:00:34.055681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055409    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd" containerName="cleanup"
Oct 30 22:00:34.055681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055420    5087 cpu_manager.go:414] "containerMap: removing container" podUID="c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd" containerName="cleanup"
Oct 30 22:00:34.055681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055426    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd" containerName="cleanup"
Oct 30 22:00:34.055681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055438    5087 cpu_manager.go:414] "containerMap: removing container" podUID="e8492282-c9d5-451f-b7ae-0a037a7e3442" containerName="cleanup"
Oct 30 22:00:34.055681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055445    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8492282-c9d5-451f-b7ae-0a037a7e3442" containerName="cleanup"
Oct 30 22:00:34.055681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055469    5087 cpu_manager.go:414] "containerMap: removing container" podUID="7981f8e3-b85e-4e26-8c57-21119cdd10fd" containerName="cleanup"
Oct 30 22:00:34.055681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055475    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7981f8e3-b85e-4e26-8c57-21119cdd10fd" containerName="cleanup"
Oct 30 22:00:34.055681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055640    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="4688f9ff-10a5-45ff-9117-8a5bbd5adf37" containerName="cleanup"
Oct 30 22:00:34.055681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055655    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd" containerName="cleanup"
Oct 30 22:00:34.055681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055664    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="bccc6aee-c25a-4d9f-8d5c-32f417307675" containerName="cleanup"
Oct 30 22:00:34.055681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055674    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8492282-c9d5-451f-b7ae-0a037a7e3442" containerName="cleanup"
Oct 30 22:00:34.055681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055686    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="3176565c-81c7-4652-8b43-8f5ab5aa6f08" containerName="cleanup"
Oct 30 22:00:34.055681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055698    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="bccc6aee-c25a-4d9f-8d5c-32f417307675" containerName="cleanup"
Oct 30 22:00:34.057078 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055706    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="7981f8e3-b85e-4e26-8c57-21119cdd10fd" containerName="cleanup"
Oct 30 22:00:34.057078 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055715    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6ac33a4-d857-4194-97ae-a43c681cbf43" containerName="revoke-keys"
Oct 30 22:00:34.057078 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055722    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8492282-c9d5-451f-b7ae-0a037a7e3442" containerName="cleanup"
Oct 30 22:00:34.057078 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055732    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="3176565c-81c7-4652-8b43-8f5ab5aa6f08" containerName="cleanup"
Oct 30 22:00:34.057078 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055739    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd" containerName="cleanup"
Oct 30 22:00:34.057078 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055748    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c0c03c-d543-4959-b0ac-6c87fb1ee8cd" containerName="cleanup"
Oct 30 22:00:34.057078 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055755    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8492282-c9d5-451f-b7ae-0a037a7e3442" containerName="cleanup"
Oct 30 22:00:34.057078 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055764    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="7981f8e3-b85e-4e26-8c57-21119cdd10fd" containerName="cleanup"
Oct 30 22:00:34.057078 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.055774    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="4688f9ff-10a5-45ff-9117-8a5bbd5adf37" containerName="cleanup"
Oct 30 22:00:34.062270 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.062249    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5966c85-72dkf"]
Oct 30 22:00:34.062414 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.062388    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caaa84af-7s79k"
Oct 30 22:00:34.102799 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.102770    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5966c85-72dkf"]
Oct 30 22:00:34.115710 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.115686    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:00:34.117116 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.117096    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caaa84af-7s79k"]
Oct 30 22:00:34.147525 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.147499    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:00:34.200606 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.200575    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60216f24-7193-4fd9-86a5-92a1a9dd8f92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-caaa84af-7s79k\" (UID: \"60216f24-7193-4fd9-86a5-92a1a9dd8f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caaa84af-7s79k"
Oct 30 22:00:34.200827 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.200805    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mml78\" (UniqueName: \"kubernetes.io/projected/60216f24-7193-4fd9-86a5-92a1a9dd8f92-kube-api-access-mml78\") pod \"maas-api-revoke-sub-caaa84af-7s79k\" (UID: \"60216f24-7193-4fd9-86a5-92a1a9dd8f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caaa84af-7s79k"
Oct 30 22:00:34.200872 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.200858    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60216f24-7193-4fd9-86a5-92a1a9dd8f92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-caaa84af-7s79k\" (UID: \"60216f24-7193-4fd9-86a5-92a1a9dd8f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caaa84af-7s79k"
Oct 30 22:00:34.301792 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.301753    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60216f24-7193-4fd9-86a5-92a1a9dd8f92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-caaa84af-7s79k\" (UID: \"60216f24-7193-4fd9-86a5-92a1a9dd8f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caaa84af-7s79k"
Oct 30 22:00:34.301984 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.301936    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mml78\" (UniqueName: \"kubernetes.io/projected/60216f24-7193-4fd9-86a5-92a1a9dd8f92-kube-api-access-mml78\") pod \"maas-api-revoke-sub-caaa84af-7s79k\" (UID: \"60216f24-7193-4fd9-86a5-92a1a9dd8f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caaa84af-7s79k"
Oct 30 22:00:34.302060 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.301989    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60216f24-7193-4fd9-86a5-92a1a9dd8f92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-caaa84af-7s79k\" (UID: \"60216f24-7193-4fd9-86a5-92a1a9dd8f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caaa84af-7s79k"
Oct 30 22:00:34.302394 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.302365    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60216f24-7193-4fd9-86a5-92a1a9dd8f92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-caaa84af-7s79k\" (UID: \"60216f24-7193-4fd9-86a5-92a1a9dd8f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caaa84af-7s79k"
Oct 30 22:00:34.304490 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.304445    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60216f24-7193-4fd9-86a5-92a1a9dd8f92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-caaa84af-7s79k\" (UID: \"60216f24-7193-4fd9-86a5-92a1a9dd8f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caaa84af-7s79k"
Oct 30 22:00:34.360083 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.360054    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mml78\" (UniqueName: \"kubernetes.io/projected/60216f24-7193-4fd9-86a5-92a1a9dd8f92-kube-api-access-mml78\") pod \"maas-api-revoke-sub-caaa84af-7s79k\" (UID: \"60216f24-7193-4fd9-86a5-92a1a9dd8f92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caaa84af-7s79k"
Oct 30 22:00:34.377059 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.377036    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caaa84af-7s79k"
Oct 30 22:00:34.779151 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:34.779111    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caaa84af-7s79k"]
Oct 30 22:00:35.418709 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:35.418618    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caaa84af-7s79k" event={"ID":"60216f24-7193-4fd9-86a5-92a1a9dd8f92","Type":"ContainerStarted","Data":"786a37f9508f8b8ba28ab4d1eb8e091e161d9c5ce7239f1d3f130bc9b97e9c8b"}
Oct 30 22:00:35.418709 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:35.418680    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caaa84af-7s79k" event={"ID":"60216f24-7193-4fd9-86a5-92a1a9dd8f92","Type":"ContainerStarted","Data":"16075f8c492668ba9207f1f85e047a807720f66254627d0541c7ffbda1b8ed64"}
Oct 30 22:00:35.540115 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:35.540050    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caaa84af-7s79k" podStartSLOduration=1.540030929 podStartE2EDuration="1.540030929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:35.53455297 +0000 UTC m=+1630.009056951" watchObservedRunningTime="2026-10-30 22:00:35.540030929 +0000 UTC m=+1630.014534909"
Oct 30 22:00:35.724281 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:35.724198    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9dd60c59-afc5-41f7-b751-121f62de5d74" path="/var/lib/kubelet/pods/9dd60c59-afc5-41f7-b751-121f62de5d74/volumes"
Oct 30 22:00:36.076930 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.076904    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 22:00:36.129737 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.129706    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-istio-data\") pod \"3f71bde2-938d-4994-8a4e-6f37bbbee412\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") "
Oct 30 22:00:36.129930 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.129818    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-workload-certs\") pod \"3f71bde2-938d-4994-8a4e-6f37bbbee412\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") "
Oct 30 22:00:36.130007 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.129952    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-workload-socket\") pod \"3f71bde2-938d-4994-8a4e-6f37bbbee412\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") "
Oct 30 22:00:36.130073 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.130013    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-istio-data" pod "3f71bde2-938d-4994-8a4e-6f37bbbee412" (UID: "3f71bde2-938d-4994-8a4e-6f37bbbee412"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:36.130073 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.130025    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f71bde2-938d-4994-8a4e-6f37bbbee412-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f71bde2-938d-4994-8a4e-6f37bbbee412-istiod-ca-cert\") pod \"3f71bde2-938d-4994-8a4e-6f37bbbee412\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") "
Oct 30 22:00:36.130073 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.130064    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f71bde2-938d-4994-8a4e-6f37bbbee412-istio-token\" (UniqueName: \"kubernetes.io/projected/3f71bde2-938d-4994-8a4e-6f37bbbee412-istio-token\") pod \"3f71bde2-938d-4994-8a4e-6f37bbbee412\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") "
Oct 30 22:00:36.130231 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.130113    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-credential-socket\") pod \"3f71bde2-938d-4994-8a4e-6f37bbbee412\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") "
Oct 30 22:00:36.130231 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.130107    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-workload-certs" pod "3f71bde2-938d-4994-8a4e-6f37bbbee412" (UID: "3f71bde2-938d-4994-8a4e-6f37bbbee412"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:36.130339 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.130211    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-workload-socket" pod "3f71bde2-938d-4994-8a4e-6f37bbbee412" (UID: "3f71bde2-938d-4994-8a4e-6f37bbbee412"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:36.130339 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.130264    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f71bde2-938d-4994-8a4e-6f37bbbee412-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f71bde2-938d-4994-8a4e-6f37bbbee412-istio-podinfo\") pod \"3f71bde2-938d-4994-8a4e-6f37bbbee412\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") "
Oct 30 22:00:36.130339 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.130315    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f71bde2-938d-4994-8a4e-6f37bbbee412-kube-api-access-9d4rp\" (UniqueName: \"kubernetes.io/projected/3f71bde2-938d-4994-8a4e-6f37bbbee412-kube-api-access-9d4rp\") pod \"3f71bde2-938d-4994-8a4e-6f37bbbee412\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") "
Oct 30 22:00:36.130339 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.130334    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f71bde2-938d-4994-8a4e-6f37bbbee412-istiod-ca-cert" pod "3f71bde2-938d-4994-8a4e-6f37bbbee412" (UID: "3f71bde2-938d-4994-8a4e-6f37bbbee412"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:36.130339 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.130375    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-istio-envoy\") pod \"3f71bde2-938d-4994-8a4e-6f37bbbee412\" (UID: \"3f71bde2-938d-4994-8a4e-6f37bbbee412\") "
Oct 30 22:00:36.130643 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.130338    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-credential-socket" pod "3f71bde2-938d-4994-8a4e-6f37bbbee412" (UID: "3f71bde2-938d-4994-8a4e-6f37bbbee412"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:36.130912 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.130894    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:36.130988 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.130922    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f71bde2-938d-4994-8a4e-6f37bbbee412-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:36.130988 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.130936    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:36.130988 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.130950    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:36.130988 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.130963    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:36.132627 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.132556    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f71bde2-938d-4994-8a4e-6f37bbbee412-istio-podinfo" pod "3f71bde2-938d-4994-8a4e-6f37bbbee412" (UID: "3f71bde2-938d-4994-8a4e-6f37bbbee412"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:00:36.132627 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.132606    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f71bde2-938d-4994-8a4e-6f37bbbee412-kube-api-access-9d4rp" pod "3f71bde2-938d-4994-8a4e-6f37bbbee412" (UID: "3f71bde2-938d-4994-8a4e-6f37bbbee412"). InnerVolumeSpecName "kube-api-access-9d4rp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:36.132927 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.132906    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-istio-envoy" pod "3f71bde2-938d-4994-8a4e-6f37bbbee412" (UID: "3f71bde2-938d-4994-8a4e-6f37bbbee412"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:00:36.133104 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.133090    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f71bde2-938d-4994-8a4e-6f37bbbee412-istio-token" pod "3f71bde2-938d-4994-8a4e-6f37bbbee412" (UID: "3f71bde2-938d-4994-8a4e-6f37bbbee412"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:36.232652 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.232608    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f71bde2-938d-4994-8a4e-6f37bbbee412-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:36.232652 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.232651    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f71bde2-938d-4994-8a4e-6f37bbbee412-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:36.232850 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.232669    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d4rp\" (UniqueName: \"kubernetes.io/projected/3f71bde2-938d-4994-8a4e-6f37bbbee412-kube-api-access-9d4rp\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:36.232850 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.232685    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f71bde2-938d-4994-8a4e-6f37bbbee412-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:36.430981 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.430945    5087 generic.go:356] "Generic (PLEG): container finished" podID="3f71bde2-938d-4994-8a4e-6f37bbbee412" containerID="8f86807b09a71d7b481cb4f47603445bf891eab4df9022dd164870c32144998f" exitCode=0
Oct 30 22:00:36.430981 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.430988    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"
Oct 30 22:00:36.431541 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.431021    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd" event={"ID":"3f71bde2-938d-4994-8a4e-6f37bbbee412","Type":"ContainerDied","Data":"8f86807b09a71d7b481cb4f47603445bf891eab4df9022dd164870c32144998f"}
Oct 30 22:00:36.431541 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.431072    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd" event={"ID":"3f71bde2-938d-4994-8a4e-6f37bbbee412","Type":"ContainerDied","Data":"7c6775d1026fffc5ddf8dea4b4f0785bc332adf33ca021e86e0cd2ce034a44f2"}
Oct 30 22:00:36.431541 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.431103    5087 scope.go:122] "RemoveContainer" containerID="8f86807b09a71d7b481cb4f47603445bf891eab4df9022dd164870c32144998f"
Oct 30 22:00:36.433233 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.433208    5087 generic.go:356] "Generic (PLEG): container finished" podID="60216f24-7193-4fd9-86a5-92a1a9dd8f92" containerID="786a37f9508f8b8ba28ab4d1eb8e091e161d9c5ce7239f1d3f130bc9b97e9c8b" exitCode=0
Oct 30 22:00:36.433393 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.433301    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caaa84af-7s79k" event={"ID":"60216f24-7193-4fd9-86a5-92a1a9dd8f92","Type":"ContainerDied","Data":"786a37f9508f8b8ba28ab4d1eb8e091e161d9c5ce7239f1d3f130bc9b97e9c8b"}
Oct 30 22:00:36.442933 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.442911    5087 scope.go:122] "RemoveContainer" containerID="8f86807b09a71d7b481cb4f47603445bf891eab4df9022dd164870c32144998f"
Oct 30 22:00:36.443259 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:00:36.443241    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f86807b09a71d7b481cb4f47603445bf891eab4df9022dd164870c32144998f\": container with ID starting with 8f86807b09a71d7b481cb4f47603445bf891eab4df9022dd164870c32144998f not found: ID does not exist" containerID="8f86807b09a71d7b481cb4f47603445bf891eab4df9022dd164870c32144998f"
Oct 30 22:00:36.443311 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.443269    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f86807b09a71d7b481cb4f47603445bf891eab4df9022dd164870c32144998f"} err="failed to get container status \"8f86807b09a71d7b481cb4f47603445bf891eab4df9022dd164870c32144998f\": rpc error: code = NotFound desc = could not find container \"8f86807b09a71d7b481cb4f47603445bf891eab4df9022dd164870c32144998f\": container with ID starting with 8f86807b09a71d7b481cb4f47603445bf891eab4df9022dd164870c32144998f not found: ID does not exist"
Oct 30 22:00:36.600833 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.600748    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"]
Oct 30 22:00:36.605318 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:36.605295    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e585c53f-openshift-default-6b94747c57-5n6cd"]
Oct 30 22:00:37.631417 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:37.631394    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caaa84af-7s79k"
Oct 30 22:00:37.655585 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:37.655559    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60216f24-7193-4fd9-86a5-92a1a9dd8f92-kube-api-access-mml78\" (UniqueName: \"kubernetes.io/projected/60216f24-7193-4fd9-86a5-92a1a9dd8f92-kube-api-access-mml78\") pod \"60216f24-7193-4fd9-86a5-92a1a9dd8f92\" (UID: \"60216f24-7193-4fd9-86a5-92a1a9dd8f92\") "
Oct 30 22:00:37.655758 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:37.655742    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60216f24-7193-4fd9-86a5-92a1a9dd8f92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60216f24-7193-4fd9-86a5-92a1a9dd8f92-maas-api-xks-ca\") pod \"60216f24-7193-4fd9-86a5-92a1a9dd8f92\" (UID: \"60216f24-7193-4fd9-86a5-92a1a9dd8f92\") "
Oct 30 22:00:37.655829 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:37.655817    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60216f24-7193-4fd9-86a5-92a1a9dd8f92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60216f24-7193-4fd9-86a5-92a1a9dd8f92-maas-api-openshift-ca\") pod \"60216f24-7193-4fd9-86a5-92a1a9dd8f92\" (UID: \"60216f24-7193-4fd9-86a5-92a1a9dd8f92\") "
Oct 30 22:00:37.656228 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:37.656196    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60216f24-7193-4fd9-86a5-92a1a9dd8f92-maas-api-openshift-ca" pod "60216f24-7193-4fd9-86a5-92a1a9dd8f92" (UID: "60216f24-7193-4fd9-86a5-92a1a9dd8f92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:37.656892 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:37.656867    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60216f24-7193-4fd9-86a5-92a1a9dd8f92-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:37.657907 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:37.657879    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60216f24-7193-4fd9-86a5-92a1a9dd8f92-maas-api-xks-ca" pod "60216f24-7193-4fd9-86a5-92a1a9dd8f92" (UID: "60216f24-7193-4fd9-86a5-92a1a9dd8f92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:00:37.658000 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:37.657964    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60216f24-7193-4fd9-86a5-92a1a9dd8f92-kube-api-access-mml78" pod "60216f24-7193-4fd9-86a5-92a1a9dd8f92" (UID: "60216f24-7193-4fd9-86a5-92a1a9dd8f92"). InnerVolumeSpecName "kube-api-access-mml78". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:37.710787 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:37.710728    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f71bde2-938d-4994-8a4e-6f37bbbee412" path="/var/lib/kubelet/pods/3f71bde2-938d-4994-8a4e-6f37bbbee412/volumes"
Oct 30 22:00:37.758690 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:37.758660    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mml78\" (UniqueName: \"kubernetes.io/projected/60216f24-7193-4fd9-86a5-92a1a9dd8f92-kube-api-access-mml78\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:37.758690 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:37.758691    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60216f24-7193-4fd9-86a5-92a1a9dd8f92-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:00:38.454468 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:38.454430    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caaa84af-7s79k" event={"ID":"60216f24-7193-4fd9-86a5-92a1a9dd8f92","Type":"ContainerDied","Data":"16075f8c492668ba9207f1f85e047a807720f66254627d0541c7ffbda1b8ed64"}
Oct 30 22:00:38.454649 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:38.454521    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16075f8c492668ba9207f1f85e047a807720f66254627d0541c7ffbda1b8ed64"
Oct 30 22:00:38.454649 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:38.454600    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caaa84af-7s79k"
Oct 30 22:00:45.275154 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.275113    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"]
Oct 30 22:00:45.277788 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.277759    5087 cpu_manager.go:414] "containerMap: removing container" podUID="bccc6aee-c25a-4d9f-8d5c-32f417307675" containerName="cleanup"
Oct 30 22:00:45.277961 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.277946    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bccc6aee-c25a-4d9f-8d5c-32f417307675" containerName="cleanup"
Oct 30 22:00:45.278079 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.278067    5087 cpu_manager.go:414] "containerMap: removing container" podUID="3f71bde2-938d-4994-8a4e-6f37bbbee412" containerName="istio-proxy"
Oct 30 22:00:45.278171 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.278160    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f71bde2-938d-4994-8a4e-6f37bbbee412" containerName="istio-proxy"
Oct 30 22:00:45.278280 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.278269    5087 cpu_manager.go:414] "containerMap: removing container" podUID="60216f24-7193-4fd9-86a5-92a1a9dd8f92" containerName="revoke-keys"
Oct 30 22:00:45.278408 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.278394    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60216f24-7193-4fd9-86a5-92a1a9dd8f92" containerName="revoke-keys"
Oct 30 22:00:45.278531 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.278519    5087 cpu_manager.go:414] "containerMap: removing container" podUID="7981f8e3-b85e-4e26-8c57-21119cdd10fd" containerName="cleanup"
Oct 30 22:00:45.278623 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.278612    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7981f8e3-b85e-4e26-8c57-21119cdd10fd" containerName="cleanup"
Oct 30 22:00:45.278919 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.278900    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="4688f9ff-10a5-45ff-9117-8a5bbd5adf37" containerName="cleanup"
Oct 30 22:00:45.279034 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.279022    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f71bde2-938d-4994-8a4e-6f37bbbee412" containerName="istio-proxy"
Oct 30 22:00:45.279141 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.279130    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="60216f24-7193-4fd9-86a5-92a1a9dd8f92" containerName="revoke-keys"
Oct 30 22:00:45.279241 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.279230    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="bccc6aee-c25a-4d9f-8d5c-32f417307675" containerName="cleanup"
Oct 30 22:00:45.279337 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.279326    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="7981f8e3-b85e-4e26-8c57-21119cdd10fd" containerName="cleanup"
Oct 30 22:00:45.303600 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.303576    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"
Oct 30 22:00:45.416012 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.415980    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 22:00:45.463425 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.424505    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 22:00:45.504381 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.432859    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4tfh8\""
Oct 30 22:00:45.504629 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.432910    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 22:00:45.504759 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.451436    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/625ae489-24f8-4985-a6d8-6b3c2953a8af-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zctbt\" (UID: \"625ae489-24f8-4985-a6d8-6b3c2953a8af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"
Oct 30 22:00:45.505064 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.505040    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/625ae489-24f8-4985-a6d8-6b3c2953a8af-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zctbt\" (UID: \"625ae489-24f8-4985-a6d8-6b3c2953a8af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"
Oct 30 22:00:45.505260 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.505242    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/625ae489-24f8-4985-a6d8-6b3c2953a8af-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zctbt\" (UID: \"625ae489-24f8-4985-a6d8-6b3c2953a8af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"
Oct 30 22:00:45.505449 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.505428    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625ae489-24f8-4985-a6d8-6b3c2953a8af-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zctbt\" (UID: \"625ae489-24f8-4985-a6d8-6b3c2953a8af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"
Oct 30 22:00:45.576526 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.576433    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/625ae489-24f8-4985-a6d8-6b3c2953a8af-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zctbt\" (UID: \"625ae489-24f8-4985-a6d8-6b3c2953a8af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"
Oct 30 22:00:45.576893 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.576866    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jtj9\" (UniqueName: \"kubernetes.io/projected/625ae489-24f8-4985-a6d8-6b3c2953a8af-kube-api-access-7jtj9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zctbt\" (UID: \"625ae489-24f8-4985-a6d8-6b3c2953a8af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"
Oct 30 22:00:45.577096 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.577077    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/625ae489-24f8-4985-a6d8-6b3c2953a8af-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zctbt\" (UID: \"625ae489-24f8-4985-a6d8-6b3c2953a8af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"
Oct 30 22:00:45.695411 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.660026    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"]
Oct 30 22:00:45.767380 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.743815    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/625ae489-24f8-4985-a6d8-6b3c2953a8af-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zctbt\" (UID: \"625ae489-24f8-4985-a6d8-6b3c2953a8af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"
Oct 30 22:00:45.767380 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.743924    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/625ae489-24f8-4985-a6d8-6b3c2953a8af-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zctbt\" (UID: \"625ae489-24f8-4985-a6d8-6b3c2953a8af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"
Oct 30 22:00:45.767380 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.744070    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/625ae489-24f8-4985-a6d8-6b3c2953a8af-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zctbt\" (UID: \"625ae489-24f8-4985-a6d8-6b3c2953a8af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"
Oct 30 22:00:45.767380 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.744139    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/625ae489-24f8-4985-a6d8-6b3c2953a8af-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zctbt\" (UID: \"625ae489-24f8-4985-a6d8-6b3c2953a8af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"
Oct 30 22:00:45.767380 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.744192    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625ae489-24f8-4985-a6d8-6b3c2953a8af-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zctbt\" (UID: \"625ae489-24f8-4985-a6d8-6b3c2953a8af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"
Oct 30 22:00:45.767380 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.744223    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/625ae489-24f8-4985-a6d8-6b3c2953a8af-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zctbt\" (UID: \"625ae489-24f8-4985-a6d8-6b3c2953a8af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"
Oct 30 22:00:45.767380 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.744380    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jtj9\" (UniqueName: \"kubernetes.io/projected/625ae489-24f8-4985-a6d8-6b3c2953a8af-kube-api-access-7jtj9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zctbt\" (UID: \"625ae489-24f8-4985-a6d8-6b3c2953a8af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"
Oct 30 22:00:45.929049 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:45.928972    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625ae489-24f8-4985-a6d8-6b3c2953a8af-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zctbt\" (UID: \"625ae489-24f8-4985-a6d8-6b3c2953a8af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"
Oct 30 22:00:46.010372 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:46.005825    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/625ae489-24f8-4985-a6d8-6b3c2953a8af-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zctbt\" (UID: \"625ae489-24f8-4985-a6d8-6b3c2953a8af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"
Oct 30 22:00:46.071379 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:46.063707    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/625ae489-24f8-4985-a6d8-6b3c2953a8af-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zctbt\" (UID: \"625ae489-24f8-4985-a6d8-6b3c2953a8af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"
Oct 30 22:00:46.122408 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:46.072129    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/625ae489-24f8-4985-a6d8-6b3c2953a8af-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zctbt\" (UID: \"625ae489-24f8-4985-a6d8-6b3c2953a8af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"
Oct 30 22:00:46.122674 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:46.098116    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/625ae489-24f8-4985-a6d8-6b3c2953a8af-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zctbt\" (UID: \"625ae489-24f8-4985-a6d8-6b3c2953a8af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"
Oct 30 22:00:46.122802 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:46.121826    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/625ae489-24f8-4985-a6d8-6b3c2953a8af-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zctbt\" (UID: \"625ae489-24f8-4985-a6d8-6b3c2953a8af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"
Oct 30 22:00:46.148499 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:46.148457    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:00:46.166373 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:46.166305    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jtj9\" (UniqueName: \"kubernetes.io/projected/625ae489-24f8-4985-a6d8-6b3c2953a8af-kube-api-access-7jtj9\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zctbt\" (UID: \"625ae489-24f8-4985-a6d8-6b3c2953a8af\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"
Oct 30 22:00:46.255190 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:46.255153    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"
Oct 30 22:00:46.422791 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:46.420667    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:00:46.422791 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:46.421767    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": context deadline exceeded"
Oct 30 22:00:46.441373 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:46.437401    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:00:46.456603 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:46.454412    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 30 22:00:46.536409 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:46.536277    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:00:46.541090 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:46.538418    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 22:00:46.541090 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:46.538932    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Oct 30 22:00:46.676394 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:46.670706    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded"
Oct 30 22:00:46.676394 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:46.672108    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml" podUID="01fdf382-6833-463a-bed9-c7c096166819" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:00:46.945215 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:46.945120    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc" podUID="f6a9f73b-bb1f-4270-a38d-3703c46fae07" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:00:47.169025 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:00:47.168989    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:00:47.253460 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:47.253428    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"]
Oct 30 22:00:47.253648 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:00:47.253626    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625ae489_24f8_4985_a6d8_6b3c2953a8af.slice/crio-a79a404144167520fc6844c3cfe10a336e2d520da6f2ad32508fbfafde7f72df WatchSource:0}: Error finding container a79a404144167520fc6844c3cfe10a336e2d520da6f2ad32508fbfafde7f72df: Status 404 returned error can't find the container with id a79a404144167520fc6844c3cfe10a336e2d520da6f2ad32508fbfafde7f72df
Oct 30 22:00:47.999153 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:47.999116    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt" event={"ID":"625ae489-24f8-4985-a6d8-6b3c2953a8af","Type":"ContainerStarted","Data":"a79a404144167520fc6844c3cfe10a336e2d520da6f2ad32508fbfafde7f72df"}
Oct 30 22:00:51.166398 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:51.166343    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:00:51.166709 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:51.166445    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:00:52.046032 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:52.045994    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt" event={"ID":"625ae489-24f8-4985-a6d8-6b3c2953a8af","Type":"ContainerStarted","Data":"10ed95496956b5d64e77e7d3864c4e9c379b597a2ee28de0b0aea0709bb20629"}
Oct 30 22:00:52.046267 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:52.046247    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"
Oct 30 22:00:52.047783 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:52.047759    5087 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-zctbt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:00:52.047917 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:52.047830    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt" podUID="625ae489-24f8-4985-a6d8-6b3c2953a8af" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:00:52.263201 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:52.262199    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt" podStartSLOduration=3.353205599 podStartE2EDuration="7.262177724s" totalImagesPullingTime="3.908972125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:52.225931251 +0000 UTC m=+1646.700435230" watchObservedRunningTime="2026-10-30 22:00:52.262177724 +0000 UTC m=+1646.736681711"
Oct 30 22:00:53.052284 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:53.052232    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f2e657d-r7x9f"]
Oct 30 22:00:53.055954 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:53.055926    5087 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-zctbt container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:00:53.056081 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:53.055970    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt" podUID="625ae489-24f8-4985-a6d8-6b3c2953a8af" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:00:53.100589 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:53.100561    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f2e657d-r7x9f"]
Oct 30 22:00:53.710984 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:53.710949    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce82a2ff-1f32-4ac4-ac06-3777474deddc" path="/var/lib/kubelet/pods/ce82a2ff-1f32-4ac4-ac06-3777474deddc/volumes"
Oct 30 22:00:54.060678 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:54.060638    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e86549f5-g8j6m"]
Oct 30 22:00:54.104679 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:54.104642    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e86549f5-g8j6m"]
Oct 30 22:00:55.711249 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:55.711214    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae6565ae-cd62-4e3b-a95b-a658fbf78892" path="/var/lib/kubelet/pods/ae6565ae-cd62-4e3b-a95b-a658fbf78892/volumes"
Oct 30 22:00:56.055955 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:56.055928    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"
Oct 30 22:00:57.052303 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:57.052230    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd6c31e4-5lsd9"]
Oct 30 22:00:57.098770 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:57.098738    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd6c31e4-5lsd9"]
Oct 30 22:00:57.716178 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:00:57.716138    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c436ac2f-8f04-4af3-92f5-22968150af31" path="/var/lib/kubelet/pods/c436ac2f-8f04-4af3-92f5-22968150af31/volumes"
Oct 30 22:01:05.056242 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:05.056148    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6afc4d44-gdqgr"]
Oct 30 22:01:05.105744 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:05.105716    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6afc4d44-gdqgr"]
Oct 30 22:01:05.711678 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:05.711646    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30ad2bc8-ec58-4b55-8c49-869450d196f6" path="/var/lib/kubelet/pods/30ad2bc8-ec58-4b55-8c49-869450d196f6/volumes"
Oct 30 22:01:11.051247 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:11.051209    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d727205-75vqz"]
Oct 30 22:01:11.100753 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:11.100727    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d727205-75vqz"]
Oct 30 22:01:11.710559 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:11.710527    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3379ae7c-a890-424f-9b75-fae49bd41850" path="/var/lib/kubelet/pods/3379ae7c-a890-424f-9b75-fae49bd41850/volumes"
Oct 30 22:01:17.209028 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:01:17.208985    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:01:18.051158 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:18.051125    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-139b6711-vp8hl"]
Oct 30 22:01:18.100419 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:18.100387    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-139b6711-vp8hl"]
Oct 30 22:01:19.711816 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:19.711773    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b644d863-029e-4159-9259-0e72a2b021fd" path="/var/lib/kubelet/pods/b644d863-029e-4159-9259-0e72a2b021fd/volumes"
Oct 30 22:01:26.052726 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:26.052667    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e31d16af-2r9pz"]
Oct 30 22:01:26.099007 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:26.098973    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e31d16af-2r9pz"]
Oct 30 22:01:27.711073 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:27.711043    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3994e998-c96e-4c4f-8c5d-6f7f7ce895a8" path="/var/lib/kubelet/pods/3994e998-c96e-4c4f-8c5d-6f7f7ce895a8/volumes"
Oct 30 22:01:27.934308 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:27.934276    5087 scope.go:122] "RemoveContainer" containerID="1e38ecf796ea47df31caefc0278dd0e0346346057873765ce0a854da11a0afa8"
Oct 30 22:01:27.945422 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:27.945388    5087 scope.go:122] "RemoveContainer" containerID="d777cb8593adef67036ac0eaa7bbe33c7b3c9207ed1623c8787fc04547fb1984"
Oct 30 22:01:27.957476 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:27.957452    5087 scope.go:122] "RemoveContainer" containerID="fa7fc80ac40cb78aa9950255992325e4df353183f857a8ae3c6408619dda0f69"
Oct 30 22:01:27.975665 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:27.975640    5087 scope.go:122] "RemoveContainer" containerID="0bbc91104567ec028fe9bf983fa04158ad75da1016bc8f5f5f1d6a25415bb964"
Oct 30 22:01:27.988181 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:27.988162    5087 scope.go:122] "RemoveContainer" containerID="2da9ba7dbbce47ddd1616763c29917b80a64e9a0ad9f59a3c29a53be8062ec97"
Oct 30 22:01:31.202375 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:31.186619    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d076e07-4x9sn"]
Oct 30 22:01:31.358369 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:31.358315    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d076e07-4x9sn"]
Oct 30 22:01:31.776586 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:31.776543    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3c0574e-034f-4e96-80aa-51ca96218b8c" path="/var/lib/kubelet/pods/f3c0574e-034f-4e96-80aa-51ca96218b8c/volumes"
Oct 30 22:01:32.423804 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:32.423744    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:01:32.426375 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:32.425610    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:01:32.426375 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:32.425746    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:01:32.426593 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:32.426406    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:01:32.574375 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:32.571928    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main" probeResult="failure" output="Get \"https://10.132.0.88:8000/ready\": context deadline exceeded"
Oct 30 22:01:32.617482 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:32.617443    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:01:32.699434 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:32.698471    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:01:33.101431 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:33.089824    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:01:33.160373 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:33.142710    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded"
Oct 30 22:01:35.051012 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:35.050978    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5521b849-8gr5m"]
Oct 30 22:01:35.099240 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:35.099207    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5521b849-8gr5m"]
Oct 30 22:01:35.714960 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:35.714928    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630" path="/var/lib/kubelet/pods/d1ef26f0-5d2e-4ca5-8d34-bbd32bf58630/volumes"
Oct 30 22:01:37.059731 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:37.059697    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffde7781-5rs9s"]
Oct 30 22:01:37.109297 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:37.109269    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ffde7781-5rs9s"]
Oct 30 22:01:37.711457 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:37.711417    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2decf802-36b5-4585-b32d-6a917c82e0ac" path="/var/lib/kubelet/pods/2decf802-36b5-4585-b32d-6a917c82e0ac/volumes"
Oct 30 22:01:46.049048 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:46.049013    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b4645f5-mmmfg"]
Oct 30 22:01:46.096677 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:46.096647    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b4645f5-mmmfg"]
Oct 30 22:01:47.241695 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:01:47.241651    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:01:47.717149 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:01:47.717054    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40d91704-2d27-4030-b79e-74046213e9d6" path="/var/lib/kubelet/pods/40d91704-2d27-4030-b79e-74046213e9d6/volumes"
Oct 30 22:02:02.073543 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:02.073504    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18fa312c-jk4nh"]
Oct 30 22:02:02.126376 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:02.126324    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-18fa312c-jk4nh"]
Oct 30 22:02:03.711028 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:03.710995    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9b38122-a71b-4f4e-b70d-e03ad0a69b21" path="/var/lib/kubelet/pods/f9b38122-a71b-4f4e-b70d-e03ad0a69b21/volumes"
Oct 30 22:02:05.345930 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:05.345897    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf"]
Oct 30 22:02:05.351671 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:05.351647    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf"
Oct 30 22:02:05.389931 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:05.389896    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zkl7\" (UniqueName: \"kubernetes.io/projected/e44c4814-699c-4567-868b-57489fbc5f9f-kube-api-access-4zkl7\") pod \"maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf\" (UID: \"e44c4814-699c-4567-868b-57489fbc5f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf"
Oct 30 22:02:05.390086 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:05.390032    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e44c4814-699c-4567-868b-57489fbc5f9f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf\" (UID: \"e44c4814-699c-4567-868b-57489fbc5f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf"
Oct 30 22:02:05.390168 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:05.390156    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e44c4814-699c-4567-868b-57489fbc5f9f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf\" (UID: \"e44c4814-699c-4567-868b-57489fbc5f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf"
Oct 30 22:02:05.406488 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:05.406455    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf"]
Oct 30 22:02:05.450783 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:05.450746    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:02:05.450973 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:05.450861    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:02:05.491498 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:05.491462    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zkl7\" (UniqueName: \"kubernetes.io/projected/e44c4814-699c-4567-868b-57489fbc5f9f-kube-api-access-4zkl7\") pod \"maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf\" (UID: \"e44c4814-699c-4567-868b-57489fbc5f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf"
Oct 30 22:02:05.491662 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:05.491569    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e44c4814-699c-4567-868b-57489fbc5f9f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf\" (UID: \"e44c4814-699c-4567-868b-57489fbc5f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf"
Oct 30 22:02:05.491707 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:05.491696    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e44c4814-699c-4567-868b-57489fbc5f9f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf\" (UID: \"e44c4814-699c-4567-868b-57489fbc5f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf"
Oct 30 22:02:05.492392 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:05.492329    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e44c4814-699c-4567-868b-57489fbc5f9f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf\" (UID: \"e44c4814-699c-4567-868b-57489fbc5f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf"
Oct 30 22:02:05.494295 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:05.494268    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e44c4814-699c-4567-868b-57489fbc5f9f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf\" (UID: \"e44c4814-699c-4567-868b-57489fbc5f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf"
Oct 30 22:02:05.548662 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:05.548626    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zkl7\" (UniqueName: \"kubernetes.io/projected/e44c4814-699c-4567-868b-57489fbc5f9f-kube-api-access-4zkl7\") pod \"maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf\" (UID: \"e44c4814-699c-4567-868b-57489fbc5f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf"
Oct 30 22:02:05.667175 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:05.667081    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf"
Oct 30 22:02:06.049322 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:02:06.048797    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode44c4814_699c_4567_868b_57489fbc5f9f.slice/crio-2ebe710b9b1b44145c78ce5f336c7035789170bc1e918de5f27d1fd7f77a107f WatchSource:0}: Error finding container 2ebe710b9b1b44145c78ce5f336c7035789170bc1e918de5f27d1fd7f77a107f: Status 404 returned error can't find the container with id 2ebe710b9b1b44145c78ce5f336c7035789170bc1e918de5f27d1fd7f77a107f
Oct 30 22:02:06.049322 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:06.049224    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf"]
Oct 30 22:02:06.959969 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:06.959933    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf" event={"ID":"e44c4814-699c-4567-868b-57489fbc5f9f","Type":"ContainerStarted","Data":"eb9ffab2bd6d62f75f0a19f40a8207d86d078c05e2437dc0544cc4e633c17ce3"}
Oct 30 22:02:06.960402 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:06.959999    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf" event={"ID":"e44c4814-699c-4567-868b-57489fbc5f9f","Type":"ContainerStarted","Data":"2ebe710b9b1b44145c78ce5f336c7035789170bc1e918de5f27d1fd7f77a107f"}
Oct 30 22:02:07.076571 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:07.076504    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf" podStartSLOduration=2.076490557 podStartE2EDuration="2.076490557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:07.074938449 +0000 UTC m=+1721.549442428" watchObservedRunningTime="2026-10-30 22:02:07.076490557 +0000 UTC m=+1721.550994583"
Oct 30 22:02:07.971793 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:07.971750    5087 generic.go:356] "Generic (PLEG): container finished" podID="e44c4814-699c-4567-868b-57489fbc5f9f" containerID="eb9ffab2bd6d62f75f0a19f40a8207d86d078c05e2437dc0544cc4e633c17ce3" exitCode=0
Oct 30 22:02:07.972467 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:07.971831    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf" event={"ID":"e44c4814-699c-4567-868b-57489fbc5f9f","Type":"ContainerDied","Data":"eb9ffab2bd6d62f75f0a19f40a8207d86d078c05e2437dc0544cc4e633c17ce3"}
Oct 30 22:02:08.080542 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:08.080500    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98324f6c-5n98h"]
Oct 30 22:02:08.086870 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:08.086844    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98324f6c-5n98h"]
Oct 30 22:02:09.172295 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:09.172267    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf"
Oct 30 22:02:09.249112 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:09.249037    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e44c4814-699c-4567-868b-57489fbc5f9f-kube-api-access-4zkl7\" (UniqueName: \"kubernetes.io/projected/e44c4814-699c-4567-868b-57489fbc5f9f-kube-api-access-4zkl7\") pod \"e44c4814-699c-4567-868b-57489fbc5f9f\" (UID: \"e44c4814-699c-4567-868b-57489fbc5f9f\") "
Oct 30 22:02:09.249266 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:09.249154    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e44c4814-699c-4567-868b-57489fbc5f9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e44c4814-699c-4567-868b-57489fbc5f9f-maas-api-openshift-ca\") pod \"e44c4814-699c-4567-868b-57489fbc5f9f\" (UID: \"e44c4814-699c-4567-868b-57489fbc5f9f\") "
Oct 30 22:02:09.249474 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:09.249450    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e44c4814-699c-4567-868b-57489fbc5f9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e44c4814-699c-4567-868b-57489fbc5f9f-maas-api-xks-ca\") pod \"e44c4814-699c-4567-868b-57489fbc5f9f\" (UID: \"e44c4814-699c-4567-868b-57489fbc5f9f\") "
Oct 30 22:02:09.249628 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:09.249583    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e44c4814-699c-4567-868b-57489fbc5f9f-maas-api-openshift-ca" pod "e44c4814-699c-4567-868b-57489fbc5f9f" (UID: "e44c4814-699c-4567-868b-57489fbc5f9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:09.250156 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:09.250132    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e44c4814-699c-4567-868b-57489fbc5f9f-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:02:09.251457 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:09.251434    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e44c4814-699c-4567-868b-57489fbc5f9f-kube-api-access-4zkl7" pod "e44c4814-699c-4567-868b-57489fbc5f9f" (UID: "e44c4814-699c-4567-868b-57489fbc5f9f"). InnerVolumeSpecName "kube-api-access-4zkl7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:09.251851 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:09.251831    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e44c4814-699c-4567-868b-57489fbc5f9f-maas-api-xks-ca" pod "e44c4814-699c-4567-868b-57489fbc5f9f" (UID: "e44c4814-699c-4567-868b-57489fbc5f9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:02:09.352336 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:09.352298    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zkl7\" (UniqueName: \"kubernetes.io/projected/e44c4814-699c-4567-868b-57489fbc5f9f-kube-api-access-4zkl7\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:02:09.352336 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:09.352325    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e44c4814-699c-4567-868b-57489fbc5f9f-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:02:09.710621 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:09.710544    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62db9532-ad32-4f35-85dd-0ad224192a66" path="/var/lib/kubelet/pods/62db9532-ad32-4f35-85dd-0ad224192a66/volumes"
Oct 30 22:02:09.990263 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:09.990235    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf"
Oct 30 22:02:09.990445 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:09.990278    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf" event={"ID":"e44c4814-699c-4567-868b-57489fbc5f9f","Type":"ContainerDied","Data":"2ebe710b9b1b44145c78ce5f336c7035789170bc1e918de5f27d1fd7f77a107f"}
Oct 30 22:02:09.990445 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:09.990324    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ebe710b9b1b44145c78ce5f336c7035789170bc1e918de5f27d1fd7f77a107f"
Oct 30 22:02:17.170161 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:02:17.170126    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:02:17.751996 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:17.751964    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87e2c278-g9574"]
Oct 30 22:02:17.753339 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:17.753308    5087 cpu_manager.go:414] "containerMap: removing container" podUID="e44c4814-699c-4567-868b-57489fbc5f9f" containerName="revoke-keys"
Oct 30 22:02:17.753339 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:17.753340    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e44c4814-699c-4567-868b-57489fbc5f9f" containerName="revoke-keys"
Oct 30 22:02:17.753604 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:17.753557    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="e44c4814-699c-4567-868b-57489fbc5f9f" containerName="revoke-keys"
Oct 30 22:02:17.758397 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:17.758232    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87e2c278-g9574"
Oct 30 22:02:17.815938 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:17.815907    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:02:17.823333 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:17.823298    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87e2c278-g9574"]
Oct 30 22:02:17.834628 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:17.834593    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0bffa69-f8de-4190-a9b9-88b9fa0ff17b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-87e2c278-g9574\" (UID: \"b0bffa69-f8de-4190-a9b9-88b9fa0ff17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87e2c278-g9574"
Oct 30 22:02:17.835549 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:17.834863    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0bffa69-f8de-4190-a9b9-88b9fa0ff17b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-87e2c278-g9574\" (UID: \"b0bffa69-f8de-4190-a9b9-88b9fa0ff17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87e2c278-g9574"
Oct 30 22:02:17.835549 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:17.835071    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7mc4\" (UniqueName: \"kubernetes.io/projected/b0bffa69-f8de-4190-a9b9-88b9fa0ff17b-kube-api-access-c7mc4\") pod \"maas-api-revoke-keys-e2e-migrate-87e2c278-g9574\" (UID: \"b0bffa69-f8de-4190-a9b9-88b9fa0ff17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87e2c278-g9574"
Oct 30 22:02:17.854694 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:17.854663    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:02:17.936919 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:17.936886    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0bffa69-f8de-4190-a9b9-88b9fa0ff17b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-87e2c278-g9574\" (UID: \"b0bffa69-f8de-4190-a9b9-88b9fa0ff17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87e2c278-g9574"
Oct 30 22:02:17.936919 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:17.936934    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0bffa69-f8de-4190-a9b9-88b9fa0ff17b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-87e2c278-g9574\" (UID: \"b0bffa69-f8de-4190-a9b9-88b9fa0ff17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87e2c278-g9574"
Oct 30 22:02:17.937204 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:17.937110    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7mc4\" (UniqueName: \"kubernetes.io/projected/b0bffa69-f8de-4190-a9b9-88b9fa0ff17b-kube-api-access-c7mc4\") pod \"maas-api-revoke-keys-e2e-migrate-87e2c278-g9574\" (UID: \"b0bffa69-f8de-4190-a9b9-88b9fa0ff17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87e2c278-g9574"
Oct 30 22:02:17.937728 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:17.937705    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0bffa69-f8de-4190-a9b9-88b9fa0ff17b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-87e2c278-g9574\" (UID: \"b0bffa69-f8de-4190-a9b9-88b9fa0ff17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87e2c278-g9574"
Oct 30 22:02:17.939579 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:17.939559    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0bffa69-f8de-4190-a9b9-88b9fa0ff17b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-87e2c278-g9574\" (UID: \"b0bffa69-f8de-4190-a9b9-88b9fa0ff17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87e2c278-g9574"
Oct 30 22:02:17.994085 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:17.994052    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7mc4\" (UniqueName: \"kubernetes.io/projected/b0bffa69-f8de-4190-a9b9-88b9fa0ff17b-kube-api-access-c7mc4\") pod \"maas-api-revoke-keys-e2e-migrate-87e2c278-g9574\" (UID: \"b0bffa69-f8de-4190-a9b9-88b9fa0ff17b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87e2c278-g9574"
Oct 30 22:02:18.088961 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:18.088881    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87e2c278-g9574"
Oct 30 22:02:18.473570 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:18.473530    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87e2c278-g9574"]
Oct 30 22:02:18.474079 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:02:18.474051    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0bffa69_f8de_4190_a9b9_88b9fa0ff17b.slice/crio-22eb718344d5310d88a81b732d87da32721e375cde43016cd5d2b888d0d07368 WatchSource:0}: Error finding container 22eb718344d5310d88a81b732d87da32721e375cde43016cd5d2b888d0d07368: Status 404 returned error can't find the container with id 22eb718344d5310d88a81b732d87da32721e375cde43016cd5d2b888d0d07368
Oct 30 22:02:19.081952 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:19.081916    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87e2c278-g9574" event={"ID":"b0bffa69-f8de-4190-a9b9-88b9fa0ff17b","Type":"ContainerStarted","Data":"3429564b1fb68fe989502687cec26043a60068a7c9e1696376e32a1bacbfe8c2"}
Oct 30 22:02:19.082127 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:19.081974    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87e2c278-g9574" event={"ID":"b0bffa69-f8de-4190-a9b9-88b9fa0ff17b","Type":"ContainerStarted","Data":"22eb718344d5310d88a81b732d87da32721e375cde43016cd5d2b888d0d07368"}
Oct 30 22:02:19.198683 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:19.198585    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87e2c278-g9574" podStartSLOduration=2.198570961 podStartE2EDuration="2.198570961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:19.195440418 +0000 UTC m=+1733.669944397" watchObservedRunningTime="2026-10-30 22:02:19.198570961 +0000 UTC m=+1733.673074939"
Oct 30 22:02:20.092576 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:20.092535    5087 generic.go:356] "Generic (PLEG): container finished" podID="b0bffa69-f8de-4190-a9b9-88b9fa0ff17b" containerID="3429564b1fb68fe989502687cec26043a60068a7c9e1696376e32a1bacbfe8c2" exitCode=0
Oct 30 22:02:20.092973 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:20.092610    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87e2c278-g9574" event={"ID":"b0bffa69-f8de-4190-a9b9-88b9fa0ff17b","Type":"ContainerDied","Data":"3429564b1fb68fe989502687cec26043a60068a7c9e1696376e32a1bacbfe8c2"}
Oct 30 22:02:21.048056 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:21.048024    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fbb8fca-dmqnp"]
Oct 30 22:02:21.052254 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:21.052225    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fbb8fca-dmqnp"]
Oct 30 22:02:21.460206 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:21.460157    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87e2c278-g9574"
Oct 30 22:02:21.496438 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:21.496401    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0bffa69-f8de-4190-a9b9-88b9fa0ff17b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0bffa69-f8de-4190-a9b9-88b9fa0ff17b-maas-api-xks-ca\") pod \"b0bffa69-f8de-4190-a9b9-88b9fa0ff17b\" (UID: \"b0bffa69-f8de-4190-a9b9-88b9fa0ff17b\") "
Oct 30 22:02:21.496605 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:21.496585    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0bffa69-f8de-4190-a9b9-88b9fa0ff17b-kube-api-access-c7mc4\" (UniqueName: \"kubernetes.io/projected/b0bffa69-f8de-4190-a9b9-88b9fa0ff17b-kube-api-access-c7mc4\") pod \"b0bffa69-f8de-4190-a9b9-88b9fa0ff17b\" (UID: \"b0bffa69-f8de-4190-a9b9-88b9fa0ff17b\") "
Oct 30 22:02:21.496678 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:21.496660    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0bffa69-f8de-4190-a9b9-88b9fa0ff17b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0bffa69-f8de-4190-a9b9-88b9fa0ff17b-maas-api-openshift-ca\") pod \"b0bffa69-f8de-4190-a9b9-88b9fa0ff17b\" (UID: \"b0bffa69-f8de-4190-a9b9-88b9fa0ff17b\") "
Oct 30 22:02:21.497312 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:21.497218    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0bffa69-f8de-4190-a9b9-88b9fa0ff17b-maas-api-openshift-ca" pod "b0bffa69-f8de-4190-a9b9-88b9fa0ff17b" (UID: "b0bffa69-f8de-4190-a9b9-88b9fa0ff17b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:21.497500 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:21.497474    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0bffa69-f8de-4190-a9b9-88b9fa0ff17b-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:02:21.498768 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:21.498734    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0bffa69-f8de-4190-a9b9-88b9fa0ff17b-maas-api-xks-ca" pod "b0bffa69-f8de-4190-a9b9-88b9fa0ff17b" (UID: "b0bffa69-f8de-4190-a9b9-88b9fa0ff17b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:02:21.498877 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:21.498853    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0bffa69-f8de-4190-a9b9-88b9fa0ff17b-kube-api-access-c7mc4" pod "b0bffa69-f8de-4190-a9b9-88b9fa0ff17b" (UID: "b0bffa69-f8de-4190-a9b9-88b9fa0ff17b"). InnerVolumeSpecName "kube-api-access-c7mc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:21.599513 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:21.599414    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0bffa69-f8de-4190-a9b9-88b9fa0ff17b-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:02:21.599513 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:21.599465    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7mc4\" (UniqueName: \"kubernetes.io/projected/b0bffa69-f8de-4190-a9b9-88b9fa0ff17b-kube-api-access-c7mc4\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:02:21.710739 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:21.710704    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4b8006f-127c-4a1b-8019-e0c5176e8d75" path="/var/lib/kubelet/pods/d4b8006f-127c-4a1b-8019-e0c5176e8d75/volumes"
Oct 30 22:02:22.118061 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:22.118034    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87e2c278-g9574"
Oct 30 22:02:22.118232 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:22.118078    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87e2c278-g9574" event={"ID":"b0bffa69-f8de-4190-a9b9-88b9fa0ff17b","Type":"ContainerDied","Data":"22eb718344d5310d88a81b732d87da32721e375cde43016cd5d2b888d0d07368"}
Oct 30 22:02:22.118232 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:22.118159    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22eb718344d5310d88a81b732d87da32721e375cde43016cd5d2b888d0d07368"
Oct 30 22:02:28.051920 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:28.051873    5087 scope.go:122] "RemoveContainer" containerID="5eba27a2d0b257543bea6a18ef077330146272271677554e48292563fd97b8ba"
Oct 30 22:02:28.066368 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:28.063668    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f988e89a-g765q"]
Oct 30 22:02:28.083807 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:28.083436    5087 scope.go:122] "RemoveContainer" containerID="c97b0f93480e7ee36f71c718131946d665273e4df35b0e2246a021785eb0a33d"
Oct 30 22:02:28.087222 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:28.087191    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f988e89a-g765q"]
Oct 30 22:02:28.098787 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:28.098768    5087 scope.go:122] "RemoveContainer" containerID="09d5af20382a619a1f3eff52c7b9c7319105f54b6704b99f2af699a5da842844"
Oct 30 22:02:28.110470 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:28.110447    5087 scope.go:122] "RemoveContainer" containerID="bdda87aa1ec77092e81dc0a78d2aed1a90a73cc02f6978486f739b9fe1169495"
Oct 30 22:02:28.130295 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:28.130273    5087 scope.go:122] "RemoveContainer" containerID="786f538dc8fb4ec0bac9f57a072c52b8b517752d4f3a631ae388b9ca1ace3b9a"
Oct 30 22:02:28.141700 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:28.141678    5087 scope.go:122] "RemoveContainer" containerID="80f13c2d112b0d9bcf765886d6672a2b21f51c04abcda736614bdd5ba317ab2e"
Oct 30 22:02:28.166081 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:28.166056    5087 scope.go:122] "RemoveContainer" containerID="1bc6d10da06ec1e1fac735331fa5705d852a07a1c01343a6ce3a9f39accc05c2"
Oct 30 22:02:28.177047 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:28.177025    5087 scope.go:122] "RemoveContainer" containerID="ef8f6b66b20ae7932faff32d0688d08b4eeda2c08d7ec410b0b9cca260c78f86"
Oct 30 22:02:28.190282 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:28.190260    5087 scope.go:122] "RemoveContainer" containerID="685bcae5ac5a6d78af32ed17e8242a111f72b18f6df200f3b955a9c7683f6929"
Oct 30 22:02:29.729774 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:29.729722    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8d32e3b-810b-40d4-a5bd-73b119077b30" path="/var/lib/kubelet/pods/b8d32e3b-810b-40d4-a5bd-73b119077b30/volumes"
Oct 30 22:02:32.048326 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:32.048285    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-744fc02d-zlt24"]
Oct 30 22:02:32.097016 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:32.096982    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-744fc02d-zlt24"]
Oct 30 22:02:33.710600 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:33.710562    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="687b2968-74a8-4969-9689-3050cf8415a7" path="/var/lib/kubelet/pods/687b2968-74a8-4969-9689-3050cf8415a7/volumes"
Oct 30 22:02:35.054624 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:35.054406    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc252a9a-8g9l7"]
Oct 30 22:02:35.060905 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:35.060873    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc252a9a-8g9l7"]
Oct 30 22:02:35.710878 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:02:35.710846    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b52a697c-969c-4df2-84ce-80e2a399ffcb" path="/var/lib/kubelet/pods/b52a697c-969c-4df2-84ce-80e2a399ffcb/volumes"
Oct 30 22:02:47.170543 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:02:47.170507    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:03:00.057715 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:00.057677    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf"]
Oct 30 22:03:00.109208 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:00.109168    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-2b8311b7-hv8lf"]
Oct 30 22:03:00.663022 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:00.662984    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"]
Oct 30 22:03:00.663290 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:00.663262    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj" podUID="5337230d-c32c-42e6-8985-4ff157dd5b95" containerName="istio-proxy" containerID="cri-o://8e4f25f698847b880a0306ff29c4e5aa651186c574f274b82ad529a2285e1952" gracePeriod=30
Oct 30 22:03:01.724369 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:01.724226    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e44c4814-699c-4567-868b-57489fbc5f9f" path="/var/lib/kubelet/pods/e44c4814-699c-4567-868b-57489fbc5f9f/volumes"
Oct 30 22:03:03.050854 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:03.050824    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"]
Oct 30 22:03:03.051321 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:03.051030    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx" podUID="3f00bbc1-19c4-447a-9459-3443bbabb845" containerName="istio-proxy" containerID="cri-o://e381a93f014c13451aa6bb404250a935c9e75c9801b62a8ba9adfbd1d61a8892" gracePeriod=30
Oct 30 22:03:03.606853 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:03.606821    5087 patch_prober.go:29] interesting pod/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:03:03.607023 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:03.606893    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj" podUID="5337230d-c32c-42e6-8985-4ff157dd5b95" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:03:05.966848 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:05.966815    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 22:03:05.993421 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:05.993387    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5337230d-c32c-42e6-8985-4ff157dd5b95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5337230d-c32c-42e6-8985-4ff157dd5b95-istio-podinfo\") pod \"5337230d-c32c-42e6-8985-4ff157dd5b95\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") "
Oct 30 22:03:05.993421 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:05.993435    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-workload-socket\") pod \"5337230d-c32c-42e6-8985-4ff157dd5b95\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") "
Oct 30 22:03:05.993649 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:05.993460    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5337230d-c32c-42e6-8985-4ff157dd5b95-kube-api-access-6nr7t\" (UniqueName: \"kubernetes.io/projected/5337230d-c32c-42e6-8985-4ff157dd5b95-kube-api-access-6nr7t\") pod \"5337230d-c32c-42e6-8985-4ff157dd5b95\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") "
Oct 30 22:03:05.993649 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:05.993635    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5337230d-c32c-42e6-8985-4ff157dd5b95-istio-token\" (UniqueName: \"kubernetes.io/projected/5337230d-c32c-42e6-8985-4ff157dd5b95-istio-token\") pod \"5337230d-c32c-42e6-8985-4ff157dd5b95\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") "
Oct 30 22:03:05.993755 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:05.993681    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-workload-certs\") pod \"5337230d-c32c-42e6-8985-4ff157dd5b95\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") "
Oct 30 22:03:05.993755 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:05.993684    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-workload-socket" pod "5337230d-c32c-42e6-8985-4ff157dd5b95" (UID: "5337230d-c32c-42e6-8985-4ff157dd5b95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:05.993755 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:05.993743    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-credential-socket\") pod \"5337230d-c32c-42e6-8985-4ff157dd5b95\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") "
Oct 30 22:03:05.993928 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:05.993832    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-istio-data\") pod \"5337230d-c32c-42e6-8985-4ff157dd5b95\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") "
Oct 30 22:03:05.994022 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:05.994002    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5337230d-c32c-42e6-8985-4ff157dd5b95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5337230d-c32c-42e6-8985-4ff157dd5b95-istiod-ca-cert\") pod \"5337230d-c32c-42e6-8985-4ff157dd5b95\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") "
Oct 30 22:03:05.994131 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:05.994005    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-workload-certs" pod "5337230d-c32c-42e6-8985-4ff157dd5b95" (UID: "5337230d-c32c-42e6-8985-4ff157dd5b95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:05.994194 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:05.994141    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-istio-envoy\") pod \"5337230d-c32c-42e6-8985-4ff157dd5b95\" (UID: \"5337230d-c32c-42e6-8985-4ff157dd5b95\") "
Oct 30 22:03:05.994255 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:05.994031    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-credential-socket" pod "5337230d-c32c-42e6-8985-4ff157dd5b95" (UID: "5337230d-c32c-42e6-8985-4ff157dd5b95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:05.994255 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:05.994211    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-istio-data" pod "5337230d-c32c-42e6-8985-4ff157dd5b95" (UID: "5337230d-c32c-42e6-8985-4ff157dd5b95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:05.994579 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:05.994380    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5337230d-c32c-42e6-8985-4ff157dd5b95-istiod-ca-cert" pod "5337230d-c32c-42e6-8985-4ff157dd5b95" (UID: "5337230d-c32c-42e6-8985-4ff157dd5b95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:05.995079 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:05.995054    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5337230d-c32c-42e6-8985-4ff157dd5b95-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:05.995192 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:05.995085    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:05.995192 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:05.995102    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:05.995192 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:05.995117    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:05.995192 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:05.995132    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:05.996053 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:05.996029    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5337230d-c32c-42e6-8985-4ff157dd5b95-istio-podinfo" pod "5337230d-c32c-42e6-8985-4ff157dd5b95" (UID: "5337230d-c32c-42e6-8985-4ff157dd5b95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:03:05.996053 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:05.996038    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5337230d-c32c-42e6-8985-4ff157dd5b95-istio-token" pod "5337230d-c32c-42e6-8985-4ff157dd5b95" (UID: "5337230d-c32c-42e6-8985-4ff157dd5b95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:05.996308 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:05.996292    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5337230d-c32c-42e6-8985-4ff157dd5b95-kube-api-access-6nr7t" pod "5337230d-c32c-42e6-8985-4ff157dd5b95" (UID: "5337230d-c32c-42e6-8985-4ff157dd5b95"). InnerVolumeSpecName "kube-api-access-6nr7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:05.996398 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:05.996338    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-istio-envoy" pod "5337230d-c32c-42e6-8985-4ff157dd5b95" (UID: "5337230d-c32c-42e6-8985-4ff157dd5b95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:06.049538 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:06.049508    5087 generic.go:356] "Generic (PLEG): container finished" podID="5337230d-c32c-42e6-8985-4ff157dd5b95" containerID="8e4f25f698847b880a0306ff29c4e5aa651186c574f274b82ad529a2285e1952" exitCode=0
Oct 30 22:03:06.049725 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:06.049544    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj" event={"ID":"5337230d-c32c-42e6-8985-4ff157dd5b95","Type":"ContainerDied","Data":"8e4f25f698847b880a0306ff29c4e5aa651186c574f274b82ad529a2285e1952"}
Oct 30 22:03:06.049725 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:06.049562    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"
Oct 30 22:03:06.049725 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:06.049585    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj" event={"ID":"5337230d-c32c-42e6-8985-4ff157dd5b95","Type":"ContainerDied","Data":"b14b5a8e259fb0d4fee48368aff3f1e16bf108fd61b3ecf01ecc3369ca53999f"}
Oct 30 22:03:06.049725 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:06.049606    5087 scope.go:122] "RemoveContainer" containerID="8e4f25f698847b880a0306ff29c4e5aa651186c574f274b82ad529a2285e1952"
Oct 30 22:03:06.062030 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:06.062009    5087 scope.go:122] "RemoveContainer" containerID="8e4f25f698847b880a0306ff29c4e5aa651186c574f274b82ad529a2285e1952"
Oct 30 22:03:06.062394 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:03:06.062337    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e4f25f698847b880a0306ff29c4e5aa651186c574f274b82ad529a2285e1952\": container with ID starting with 8e4f25f698847b880a0306ff29c4e5aa651186c574f274b82ad529a2285e1952 not found: ID does not exist" containerID="8e4f25f698847b880a0306ff29c4e5aa651186c574f274b82ad529a2285e1952"
Oct 30 22:03:06.062454 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:06.062407    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e4f25f698847b880a0306ff29c4e5aa651186c574f274b82ad529a2285e1952"} err="failed to get container status \"8e4f25f698847b880a0306ff29c4e5aa651186c574f274b82ad529a2285e1952\": rpc error: code = NotFound desc = could not find container \"8e4f25f698847b880a0306ff29c4e5aa651186c574f274b82ad529a2285e1952\": container with ID starting with 8e4f25f698847b880a0306ff29c4e5aa651186c574f274b82ad529a2285e1952 not found: ID does not exist"
Oct 30 22:03:06.096690 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:06.096653    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5337230d-c32c-42e6-8985-4ff157dd5b95-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:06.096854 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:06.096695    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nr7t\" (UniqueName: \"kubernetes.io/projected/5337230d-c32c-42e6-8985-4ff157dd5b95-kube-api-access-6nr7t\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:06.096854 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:06.096716    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5337230d-c32c-42e6-8985-4ff157dd5b95-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:06.096854 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:06.096731    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5337230d-c32c-42e6-8985-4ff157dd5b95-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:06.125964 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:03:06.125931    5087 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5337230d_c32c_42e6_8985_4ff157dd5b95.slice/crio-b14b5a8e259fb0d4fee48368aff3f1e16bf108fd61b3ecf01ecc3369ca53999f\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5337230d_c32c_42e6_8985_4ff157dd5b95.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:03:06.257405 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:06.254417    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"]
Oct 30 22:03:06.262326 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:06.262297    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-msfnj"]
Oct 30 22:03:06.695583 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:06.695507    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87e2c278-g9574"]
Oct 30 22:03:06.701292 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:06.701269    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-87e2c278-g9574"]
Oct 30 22:03:07.711137 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:07.711108    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5337230d-c32c-42e6-8985-4ff157dd5b95" path="/var/lib/kubelet/pods/5337230d-c32c-42e6-8985-4ff157dd5b95/volumes"
Oct 30 22:03:07.711640 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:07.711624    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0bffa69-f8de-4190-a9b9-88b9fa0ff17b" path="/var/lib/kubelet/pods/b0bffa69-f8de-4190-a9b9-88b9fa0ff17b/volumes"
Oct 30 22:03:07.761979 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:07.761938    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:03:07.762141 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:07.762005    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx" podUID="3f00bbc1-19c4-447a-9459-3443bbabb845" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:03:08.716064 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:08.716033    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 22:03:08.796557 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:08.796464    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-workload-socket\") pod \"3f00bbc1-19c4-447a-9459-3443bbabb845\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") "
Oct 30 22:03:08.796797 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:08.796569    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f00bbc1-19c4-447a-9459-3443bbabb845-istio-token\" (UniqueName: \"kubernetes.io/projected/3f00bbc1-19c4-447a-9459-3443bbabb845-istio-token\") pod \"3f00bbc1-19c4-447a-9459-3443bbabb845\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") "
Oct 30 22:03:08.796797 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:08.796620    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f00bbc1-19c4-447a-9459-3443bbabb845-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f00bbc1-19c4-447a-9459-3443bbabb845-istio-podinfo\") pod \"3f00bbc1-19c4-447a-9459-3443bbabb845\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") "
Oct 30 22:03:08.796797 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:08.796767    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-credential-socket\") pod \"3f00bbc1-19c4-447a-9459-3443bbabb845\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") "
Oct 30 22:03:08.796974 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:08.796840    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f00bbc1-19c4-447a-9459-3443bbabb845-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f00bbc1-19c4-447a-9459-3443bbabb845-istiod-ca-cert\") pod \"3f00bbc1-19c4-447a-9459-3443bbabb845\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") "
Oct 30 22:03:08.796974 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:08.796923    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f00bbc1-19c4-447a-9459-3443bbabb845-kube-api-access-k5wn4\" (UniqueName: \"kubernetes.io/projected/3f00bbc1-19c4-447a-9459-3443bbabb845-kube-api-access-k5wn4\") pod \"3f00bbc1-19c4-447a-9459-3443bbabb845\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") "
Oct 30 22:03:08.797088 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:08.796992    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-workload-certs\") pod \"3f00bbc1-19c4-447a-9459-3443bbabb845\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") "
Oct 30 22:03:08.797088 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:08.797069    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-istio-data\") pod \"3f00bbc1-19c4-447a-9459-3443bbabb845\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") "
Oct 30 22:03:08.797195 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:08.797164    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-istio-envoy\") pod \"3f00bbc1-19c4-447a-9459-3443bbabb845\" (UID: \"3f00bbc1-19c4-447a-9459-3443bbabb845\") "
Oct 30 22:03:08.798289 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:08.796764    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-workload-socket" pod "3f00bbc1-19c4-447a-9459-3443bbabb845" (UID: "3f00bbc1-19c4-447a-9459-3443bbabb845"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:08.798436 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:08.797689    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-workload-certs" pod "3f00bbc1-19c4-447a-9459-3443bbabb845" (UID: "3f00bbc1-19c4-447a-9459-3443bbabb845"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:08.798436 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:08.797780    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f00bbc1-19c4-447a-9459-3443bbabb845-istiod-ca-cert" pod "3f00bbc1-19c4-447a-9459-3443bbabb845" (UID: "3f00bbc1-19c4-447a-9459-3443bbabb845"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:08.798436 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:08.798012    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-istio-data" pod "3f00bbc1-19c4-447a-9459-3443bbabb845" (UID: "3f00bbc1-19c4-447a-9459-3443bbabb845"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:08.798436 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:08.798249    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-credential-socket" pod "3f00bbc1-19c4-447a-9459-3443bbabb845" (UID: "3f00bbc1-19c4-447a-9459-3443bbabb845"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:08.800876 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:08.800844    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f00bbc1-19c4-447a-9459-3443bbabb845-istio-podinfo" pod "3f00bbc1-19c4-447a-9459-3443bbabb845" (UID: "3f00bbc1-19c4-447a-9459-3443bbabb845"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:03:08.801069 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:08.801006    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f00bbc1-19c4-447a-9459-3443bbabb845-istio-token" pod "3f00bbc1-19c4-447a-9459-3443bbabb845" (UID: "3f00bbc1-19c4-447a-9459-3443bbabb845"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:08.801069 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:08.801037    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-istio-envoy" pod "3f00bbc1-19c4-447a-9459-3443bbabb845" (UID: "3f00bbc1-19c4-447a-9459-3443bbabb845"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:08.802716 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:08.802697    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f00bbc1-19c4-447a-9459-3443bbabb845-kube-api-access-k5wn4" pod "3f00bbc1-19c4-447a-9459-3443bbabb845" (UID: "3f00bbc1-19c4-447a-9459-3443bbabb845"). InnerVolumeSpecName "kube-api-access-k5wn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:08.899520 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:08.899341    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:08.899520 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:08.899398    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:08.899520 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:08.899412    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:08.899520 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:08.899425    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f00bbc1-19c4-447a-9459-3443bbabb845-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:08.899520 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:08.899442    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f00bbc1-19c4-447a-9459-3443bbabb845-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:08.899520 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:08.899454    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:08.899520 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:08.899466    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f00bbc1-19c4-447a-9459-3443bbabb845-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:08.899520 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:08.899480    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5wn4\" (UniqueName: \"kubernetes.io/projected/3f00bbc1-19c4-447a-9459-3443bbabb845-kube-api-access-k5wn4\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:08.899520 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:08.899494    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f00bbc1-19c4-447a-9459-3443bbabb845-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:09.085183 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:09.085086    5087 generic.go:356] "Generic (PLEG): container finished" podID="3f00bbc1-19c4-447a-9459-3443bbabb845" containerID="e381a93f014c13451aa6bb404250a935c9e75c9801b62a8ba9adfbd1d61a8892" exitCode=0
Oct 30 22:03:09.085372 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:09.085267    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx" event={"ID":"3f00bbc1-19c4-447a-9459-3443bbabb845","Type":"ContainerDied","Data":"e381a93f014c13451aa6bb404250a935c9e75c9801b62a8ba9adfbd1d61a8892"}
Oct 30 22:03:09.085476 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:09.085390    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"
Oct 30 22:03:09.087648 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:09.087611    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx" event={"ID":"3f00bbc1-19c4-447a-9459-3443bbabb845","Type":"ContainerDied","Data":"6f94ac56b324f9a012b9d60b9e68a81f708f12c2e193ff2eff8750900b358cf5"}
Oct 30 22:03:09.087783 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:09.087668    5087 scope.go:122] "RemoveContainer" containerID="e381a93f014c13451aa6bb404250a935c9e75c9801b62a8ba9adfbd1d61a8892"
Oct 30 22:03:09.104415 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:09.104378    5087 scope.go:122] "RemoveContainer" containerID="e381a93f014c13451aa6bb404250a935c9e75c9801b62a8ba9adfbd1d61a8892"
Oct 30 22:03:09.104741 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:03:09.104722    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e381a93f014c13451aa6bb404250a935c9e75c9801b62a8ba9adfbd1d61a8892\": container with ID starting with e381a93f014c13451aa6bb404250a935c9e75c9801b62a8ba9adfbd1d61a8892 not found: ID does not exist" containerID="e381a93f014c13451aa6bb404250a935c9e75c9801b62a8ba9adfbd1d61a8892"
Oct 30 22:03:09.104807 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:09.104750    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e381a93f014c13451aa6bb404250a935c9e75c9801b62a8ba9adfbd1d61a8892"} err="failed to get container status \"e381a93f014c13451aa6bb404250a935c9e75c9801b62a8ba9adfbd1d61a8892\": rpc error: code = NotFound desc = could not find container \"e381a93f014c13451aa6bb404250a935c9e75c9801b62a8ba9adfbd1d61a8892\": container with ID starting with e381a93f014c13451aa6bb404250a935c9e75c9801b62a8ba9adfbd1d61a8892 not found: ID does not exist"
Oct 30 22:03:09.269534 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:09.269504    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"]
Oct 30 22:03:09.273053 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:09.273027    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-wn9kx"]
Oct 30 22:03:09.712581 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:09.712534    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f00bbc1-19c4-447a-9459-3443bbabb845" path="/var/lib/kubelet/pods/3f00bbc1-19c4-447a-9459-3443bbabb845/volumes"
Oct 30 22:03:12.046596 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:12.046551    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdd577bb-pjk4b"]
Oct 30 22:03:12.095741 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:12.095712    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdd577bb-pjk4b"]
Oct 30 22:03:13.718967 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:13.718926    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f9bcb50-f502-4e1b-8d09-9d607752ac38" path="/var/lib/kubelet/pods/3f9bcb50-f502-4e1b-8d09-9d607752ac38/volumes"
Oct 30 22:03:17.390439 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:03:17.390399    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:03:18.150038 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:18.150004    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"]
Oct 30 22:03:18.150231 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:18.150211    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b" podUID="09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b" containerName="istio-proxy" containerID="cri-o://b31aa2bcb35339be1dff9831eebc09071b1dc7c70b0e9f49eaef7fb2e4bbe60b" gracePeriod=30
Oct 30 22:03:23.399531 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:23.399483    5087 generic.go:356] "Generic (PLEG): container finished" podID="09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b" containerID="b31aa2bcb35339be1dff9831eebc09071b1dc7c70b0e9f49eaef7fb2e4bbe60b" exitCode=0
Oct 30 22:03:23.400039 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:23.399551    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b" event={"ID":"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b","Type":"ContainerDied","Data":"b31aa2bcb35339be1dff9831eebc09071b1dc7c70b0e9f49eaef7fb2e4bbe60b"}
Oct 30 22:03:23.444734 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:23.444676    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 22:03:23.519983 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:23.519947    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-credential-socket\") pod \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") "
Oct 30 22:03:23.520174 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:23.520077    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-workload-socket\") pod \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") "
Oct 30 22:03:23.520174 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:23.520143    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istio-envoy\") pod \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") "
Oct 30 22:03:23.520285 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:23.520191    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-kube-api-access-6ctr4\" (UniqueName: \"kubernetes.io/projected/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-kube-api-access-6ctr4\") pod \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") "
Oct 30 22:03:23.520285 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:23.520256    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istio-data\") pod \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") "
Oct 30 22:03:23.520407 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:23.520331    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istio-token\" (UniqueName: \"kubernetes.io/projected/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istio-token\") pod \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") "
Oct 30 22:03:23.520407 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:23.520342    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-credential-socket" pod "09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b" (UID: "09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:23.520511 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:23.520405    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istiod-ca-cert\") pod \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") "
Oct 30 22:03:23.520511 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:23.520468    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istio-podinfo\") pod \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") "
Oct 30 22:03:23.520609 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:23.520579    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-workload-certs\") pod \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\" (UID: \"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b\") "
Oct 30 22:03:23.521315 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:23.521292    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:23.522506 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:23.521427    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-workload-socket" pod "09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b" (UID: "09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:23.522656 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:23.522328    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-workload-certs" pod "09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b" (UID: "09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:23.522759 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:23.522250    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istiod-ca-cert" pod "09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b" (UID: "09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:23.522849 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:23.522464    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istio-data" pod "09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b" (UID: "09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:23.524949 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:23.524913    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-kube-api-access-6ctr4" pod "09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b" (UID: "09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b"). InnerVolumeSpecName "kube-api-access-6ctr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:23.525184 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:23.525155    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istio-token" pod "09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b" (UID: "09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:23.525318 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:23.525294    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istio-podinfo" pod "09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b" (UID: "09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:03:23.526247 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:23.526227    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istio-envoy" pod "09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b" (UID: "09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:03:23.645775 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:23.645728    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:23.645775 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:23.645763    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:23.645775 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:23.645778    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ctr4\" (UniqueName: \"kubernetes.io/projected/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-kube-api-access-6ctr4\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:23.646099 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:23.645793    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:23.646099 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:23.645806    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:23.646099 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:23.645817    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:23.646099 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:23.645829    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:23.646099 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:23.645840    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:03:24.412634 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:24.412593    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b" event={"ID":"09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b","Type":"ContainerDied","Data":"0d757e359f657936ce471fe84368492e9dae16c3427a6e32fea742c7e2e1d2bf"}
Oct 30 22:03:24.413129 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:24.412670    5087 scope.go:122] "RemoveContainer" containerID="b31aa2bcb35339be1dff9831eebc09071b1dc7c70b0e9f49eaef7fb2e4bbe60b"
Oct 30 22:03:24.413129 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:24.412727    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"
Oct 30 22:03:24.659380 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:24.655062    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"]
Oct 30 22:03:24.701008 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:24.699495    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-bf95b"]
Oct 30 22:03:25.710814 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:25.710783    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b" path="/var/lib/kubelet/pods/09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b/volumes"
Oct 30 22:03:25.993947 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:25.993916    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 22:03:26.007461 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:26.007435    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 22:03:26.008723 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:26.008703    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 22:03:26.017631 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:26.017615    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 22:03:28.411688 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:28.411658    5087 scope.go:122] "RemoveContainer" containerID="09da43b8edf3e0b15768d798b13b9a75652be1ba6c40a8220bc6db6e6ca40e77"
Oct 30 22:03:28.428614 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:28.428585    5087 scope.go:122] "RemoveContainer" containerID="ffc8e2e96735b708913ea06ab0648e40044d04ca863f27524207ac3fab09f10d"
Oct 30 22:03:28.439154 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:28.439133    5087 scope.go:122] "RemoveContainer" containerID="ecc77fe7500e2e06de3011141f0c3a08f834daf0dcb15d595a2babe5644a7182"
Oct 30 22:03:28.450912 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:28.450859    5087 scope.go:122] "RemoveContainer" containerID="111995d0d829bc5d38de856ca6b3fab1ce7163d08035600ce8df2a7d7c13d8f3"
Oct 30 22:03:28.464893 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:28.464869    5087 scope.go:122] "RemoveContainer" containerID="d2836016f17582522aa1dcc5fdd9edc25c6cc763821ab994c7e64e69842b47d3"
Oct 30 22:03:28.479845 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:28.479470    5087 scope.go:122] "RemoveContainer" containerID="8ee918ffae3ad632371ba5bd9ede01592baa975edaa84dba811a86498f782acd"
Oct 30 22:03:28.504980 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:03:28.504955    5087 scope.go:122] "RemoveContainer" containerID="3639c6b19a3071c56e64202b85e9cba57da9c623824606c9458e5db91c281a59"
Oct 30 22:03:47.169857 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:03:47.169819    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:04:03.531469 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:03.531438    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"]
Oct 30 22:04:03.531999 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:03.531624    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg" podUID="894d8554-651d-4ad9-8521-da6c98edca6d" containerName="istio-proxy" containerID="cri-o://44faaa6dc01eadd1454099f86f68e37eb2d12eae44ef8d995c62974668560deb" gracePeriod=30
Oct 30 22:04:03.581384 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:03.581311    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"]
Oct 30 22:04:03.581576 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:03.581555    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96" podUID="727b252d-ad3e-4e43-9fb8-471fd2957231" containerName="istio-proxy" containerID="cri-o://2a184e22927900b74b6226bf17a18ebf3d0c92e1216ff49aad0d86bc8c54e247" gracePeriod=30
Oct 30 22:04:03.585086 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:03.585051    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"]
Oct 30 22:04:03.585266 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:03.585246    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7" podUID="146b54d0-dc9b-4d22-9c30-fe25bd1d5194" containerName="istio-proxy" containerID="cri-o://92fff8afe405ce49e361b133b95ed80dd7e4ab927054e393d22924b42f9ea685" gracePeriod=30
Oct 30 22:04:03.630149 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:03.630101    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"]
Oct 30 22:04:03.630362 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:03.630315    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k" podUID="89d4443a-0fb1-461e-a866-05351817130e" containerName="istio-proxy" containerID="cri-o://a995abf6d93c7e70bb189419720531fb8bf63b9983121430844125d99718b834" gracePeriod=30
Oct 30 22:04:04.646013 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:04.645934    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"]
Oct 30 22:04:04.646433 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:04.646188    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp" podUID="aed464f0-0b79-467a-b131-fa6944d01941" containerName="istio-proxy" containerID="cri-o://a82229b4acff245f57189518bbc0d179e406e82a54c2debb702dff2a9233a7e5" gracePeriod=30
Oct 30 22:04:05.545146 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:05.545107    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:04:05.545337 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:05.545184    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7" podUID="146b54d0-dc9b-4d22-9c30-fe25bd1d5194" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:04:08.866124 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:08.865948    5087 generic.go:356] "Generic (PLEG): container finished" podID="894d8554-651d-4ad9-8521-da6c98edca6d" containerID="44faaa6dc01eadd1454099f86f68e37eb2d12eae44ef8d995c62974668560deb" exitCode=0
Oct 30 22:04:08.866124 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:08.866059    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg" event={"ID":"894d8554-651d-4ad9-8521-da6c98edca6d","Type":"ContainerDied","Data":"44faaa6dc01eadd1454099f86f68e37eb2d12eae44ef8d995c62974668560deb"}
Oct 30 22:04:08.870056 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:08.869668    5087 generic.go:356] "Generic (PLEG): container finished" podID="146b54d0-dc9b-4d22-9c30-fe25bd1d5194" containerID="92fff8afe405ce49e361b133b95ed80dd7e4ab927054e393d22924b42f9ea685" exitCode=0
Oct 30 22:04:08.870056 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:08.869740    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7" event={"ID":"146b54d0-dc9b-4d22-9c30-fe25bd1d5194","Type":"ContainerDied","Data":"92fff8afe405ce49e361b133b95ed80dd7e4ab927054e393d22924b42f9ea685"}
Oct 30 22:04:08.871898 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:08.871717    5087 generic.go:356] "Generic (PLEG): container finished" podID="89d4443a-0fb1-461e-a866-05351817130e" containerID="a995abf6d93c7e70bb189419720531fb8bf63b9983121430844125d99718b834" exitCode=0
Oct 30 22:04:08.871898 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:08.871850    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k" event={"ID":"89d4443a-0fb1-461e-a866-05351817130e","Type":"ContainerDied","Data":"a995abf6d93c7e70bb189419720531fb8bf63b9983121430844125d99718b834"}
Oct 30 22:04:08.903564 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:08.903527    5087 patch_prober.go:29] interesting pod/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body=
Oct 30 22:04:08.903787 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:08.903584    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96" podUID="727b252d-ad3e-4e43-9fb8-471fd2957231" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused"
Oct 30 22:04:09.065586 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.065556    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 22:04:09.152670 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.152629    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 22:04:09.162008 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.161982    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89d4443a-0fb1-461e-a866-05351817130e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89d4443a-0fb1-461e-a866-05351817130e-istiod-ca-cert\") pod \"89d4443a-0fb1-461e-a866-05351817130e\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") "
Oct 30 22:04:09.162253 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.162236    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-workload-certs\") pod \"89d4443a-0fb1-461e-a866-05351817130e\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") "
Oct 30 22:04:09.162712 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.162460    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89d4443a-0fb1-461e-a866-05351817130e-istiod-ca-cert" pod "89d4443a-0fb1-461e-a866-05351817130e" (UID: "89d4443a-0fb1-461e-a866-05351817130e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:09.162856 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.162602    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-workload-certs" pod "89d4443a-0fb1-461e-a866-05351817130e" (UID: "89d4443a-0fb1-461e-a866-05351817130e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:09.162943 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.162679    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89d4443a-0fb1-461e-a866-05351817130e-kube-api-access-ffn4x\" (UniqueName: \"kubernetes.io/projected/89d4443a-0fb1-461e-a866-05351817130e-kube-api-access-ffn4x\") pod \"89d4443a-0fb1-461e-a866-05351817130e\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") "
Oct 30 22:04:09.163076 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.163063    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-istio-envoy\") pod \"89d4443a-0fb1-461e-a866-05351817130e\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") "
Oct 30 22:04:09.163209 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.163197    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89d4443a-0fb1-461e-a866-05351817130e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89d4443a-0fb1-461e-a866-05351817130e-istio-podinfo\") pod \"89d4443a-0fb1-461e-a866-05351817130e\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") "
Oct 30 22:04:09.163656 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.163637    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-credential-socket\") pod \"89d4443a-0fb1-461e-a866-05351817130e\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") "
Oct 30 22:04:09.164120 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.164020    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-credential-socket" pod "89d4443a-0fb1-461e-a866-05351817130e" (UID: "89d4443a-0fb1-461e-a866-05351817130e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:09.164334 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.164320    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-workload-socket\") pod \"89d4443a-0fb1-461e-a866-05351817130e\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") "
Oct 30 22:04:09.164714 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.164609    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-workload-socket" pod "89d4443a-0fb1-461e-a866-05351817130e" (UID: "89d4443a-0fb1-461e-a866-05351817130e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:09.164823 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.164764    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-istio-data\") pod \"89d4443a-0fb1-461e-a866-05351817130e\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") "
Oct 30 22:04:09.164894 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.164842    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89d4443a-0fb1-461e-a866-05351817130e-istio-token\" (UniqueName: \"kubernetes.io/projected/89d4443a-0fb1-461e-a866-05351817130e-istio-token\") pod \"89d4443a-0fb1-461e-a866-05351817130e\" (UID: \"89d4443a-0fb1-461e-a866-05351817130e\") "
Oct 30 22:04:09.165720 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.165182    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-istio-data" pod "89d4443a-0fb1-461e-a866-05351817130e" (UID: "89d4443a-0fb1-461e-a866-05351817130e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:09.165720 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.165630    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.165720 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.165653    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89d4443a-0fb1-461e-a866-05351817130e-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.165720 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.165668    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.165720 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.165683    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.165720 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.165697    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.180734 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.169226    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-istio-envoy" pod "89d4443a-0fb1-461e-a866-05351817130e" (UID: "89d4443a-0fb1-461e-a866-05351817130e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:09.180734 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.169798    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89d4443a-0fb1-461e-a866-05351817130e-istio-token" pod "89d4443a-0fb1-461e-a866-05351817130e" (UID: "89d4443a-0fb1-461e-a866-05351817130e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:09.187915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.187881    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89d4443a-0fb1-461e-a866-05351817130e-istio-podinfo" pod "89d4443a-0fb1-461e-a866-05351817130e" (UID: "89d4443a-0fb1-461e-a866-05351817130e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:04:09.194601 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.194565    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89d4443a-0fb1-461e-a866-05351817130e-kube-api-access-ffn4x" pod "89d4443a-0fb1-461e-a866-05351817130e" (UID: "89d4443a-0fb1-461e-a866-05351817130e"). InnerVolumeSpecName "kube-api-access-ffn4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:09.215724 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.215702    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 22:04:09.217887 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.217865    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 22:04:09.267034 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.266996    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/894d8554-651d-4ad9-8521-da6c98edca6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/894d8554-651d-4ad9-8521-da6c98edca6d-istio-podinfo\") pod \"894d8554-651d-4ad9-8521-da6c98edca6d\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") "
Oct 30 22:04:09.267217 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.267065    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-credential-socket\") pod \"894d8554-651d-4ad9-8521-da6c98edca6d\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") "
Oct 30 22:04:09.267217 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.267170    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/894d8554-651d-4ad9-8521-da6c98edca6d-kube-api-access-2n4xw\" (UniqueName: \"kubernetes.io/projected/894d8554-651d-4ad9-8521-da6c98edca6d-kube-api-access-2n4xw\") pod \"894d8554-651d-4ad9-8521-da6c98edca6d\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") "
Oct 30 22:04:09.267493 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.267261    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-istio-data\") pod \"894d8554-651d-4ad9-8521-da6c98edca6d\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") "
Oct 30 22:04:09.267493 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.267312    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/894d8554-651d-4ad9-8521-da6c98edca6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/894d8554-651d-4ad9-8521-da6c98edca6d-istiod-ca-cert\") pod \"894d8554-651d-4ad9-8521-da6c98edca6d\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") "
Oct 30 22:04:09.267493 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.267397    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/894d8554-651d-4ad9-8521-da6c98edca6d-istio-token\" (UniqueName: \"kubernetes.io/projected/894d8554-651d-4ad9-8521-da6c98edca6d-istio-token\") pod \"894d8554-651d-4ad9-8521-da6c98edca6d\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") "
Oct 30 22:04:09.267661 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.267513    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-istio-envoy\") pod \"894d8554-651d-4ad9-8521-da6c98edca6d\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") "
Oct 30 22:04:09.267661 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.267567    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-workload-socket\") pod \"894d8554-651d-4ad9-8521-da6c98edca6d\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") "
Oct 30 22:04:09.267768 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.267657    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-workload-certs\") pod \"894d8554-651d-4ad9-8521-da6c98edca6d\" (UID: \"894d8554-651d-4ad9-8521-da6c98edca6d\") "
Oct 30 22:04:09.268340 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.268319    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffn4x\" (UniqueName: \"kubernetes.io/projected/89d4443a-0fb1-461e-a866-05351817130e-kube-api-access-ffn4x\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.268649 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.268627    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-istio-data" pod "894d8554-651d-4ad9-8521-da6c98edca6d" (UID: "894d8554-651d-4ad9-8521-da6c98edca6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:09.269204 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.269181    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-credential-socket" pod "894d8554-651d-4ad9-8521-da6c98edca6d" (UID: "894d8554-651d-4ad9-8521-da6c98edca6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:09.269298 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.269256    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89d4443a-0fb1-461e-a866-05351817130e-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.269298 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.269281    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89d4443a-0fb1-461e-a866-05351817130e-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.269475 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.269297    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89d4443a-0fb1-461e-a866-05351817130e-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.269544 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.269522    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-workload-certs" pod "894d8554-651d-4ad9-8521-da6c98edca6d" (UID: "894d8554-651d-4ad9-8521-da6c98edca6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:09.269689 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.269558    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/894d8554-651d-4ad9-8521-da6c98edca6d-istiod-ca-cert" pod "894d8554-651d-4ad9-8521-da6c98edca6d" (UID: "894d8554-651d-4ad9-8521-da6c98edca6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:09.269952 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.269935    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-workload-socket" pod "894d8554-651d-4ad9-8521-da6c98edca6d" (UID: "894d8554-651d-4ad9-8521-da6c98edca6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:09.271999 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.271971    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/894d8554-651d-4ad9-8521-da6c98edca6d-istio-token" pod "894d8554-651d-4ad9-8521-da6c98edca6d" (UID: "894d8554-651d-4ad9-8521-da6c98edca6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:09.272743 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.272710    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-istio-envoy" pod "894d8554-651d-4ad9-8521-da6c98edca6d" (UID: "894d8554-651d-4ad9-8521-da6c98edca6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:09.275219 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.275199    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/894d8554-651d-4ad9-8521-da6c98edca6d-kube-api-access-2n4xw" pod "894d8554-651d-4ad9-8521-da6c98edca6d" (UID: "894d8554-651d-4ad9-8521-da6c98edca6d"). InnerVolumeSpecName "kube-api-access-2n4xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:09.275460 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.275436    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/894d8554-651d-4ad9-8521-da6c98edca6d-istio-podinfo" pod "894d8554-651d-4ad9-8521-da6c98edca6d" (UID: "894d8554-651d-4ad9-8521-da6c98edca6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:04:09.370659 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.370619    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/727b252d-ad3e-4e43-9fb8-471fd2957231-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/727b252d-ad3e-4e43-9fb8-471fd2957231-istio-podinfo\") pod \"727b252d-ad3e-4e43-9fb8-471fd2957231\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") "
Oct 30 22:04:09.370847 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.370702    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-kube-api-access-tz6bj\" (UniqueName: \"kubernetes.io/projected/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-kube-api-access-tz6bj\") pod \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") "
Oct 30 22:04:09.370847 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.370743    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-credential-socket\") pod \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") "
Oct 30 22:04:09.370847 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.370778    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-workload-socket\") pod \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") "
Oct 30 22:04:09.370847 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.370841    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/727b252d-ad3e-4e43-9fb8-471fd2957231-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/727b252d-ad3e-4e43-9fb8-471fd2957231-istiod-ca-cert\") pod \"727b252d-ad3e-4e43-9fb8-471fd2957231\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") "
Oct 30 22:04:09.371071 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.370870    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-credential-socket\") pod \"727b252d-ad3e-4e43-9fb8-471fd2957231\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") "
Oct 30 22:04:09.371071 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.370906    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-istio-data\") pod \"727b252d-ad3e-4e43-9fb8-471fd2957231\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") "
Oct 30 22:04:09.371071 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.371046    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istio-envoy\") pod \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") "
Oct 30 22:04:09.371225 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.371097    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istiod-ca-cert\") pod \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") "
Oct 30 22:04:09.371225 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.371142    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istio-podinfo\") pod \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") "
Oct 30 22:04:09.371225 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.371160    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-workload-socket" pod "146b54d0-dc9b-4d22-9c30-fe25bd1d5194" (UID: "146b54d0-dc9b-4d22-9c30-fe25bd1d5194"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:09.371225 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.371185    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-workload-certs\") pod \"727b252d-ad3e-4e43-9fb8-471fd2957231\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") "
Oct 30 22:04:09.371432 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.371229    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/727b252d-ad3e-4e43-9fb8-471fd2957231-kube-api-access-dfk2t\" (UniqueName: \"kubernetes.io/projected/727b252d-ad3e-4e43-9fb8-471fd2957231-kube-api-access-dfk2t\") pod \"727b252d-ad3e-4e43-9fb8-471fd2957231\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") "
Oct 30 22:04:09.371432 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.371279    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istio-token\" (UniqueName: \"kubernetes.io/projected/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istio-token\") pod \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") "
Oct 30 22:04:09.371432 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.371400    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istio-data\") pod \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") "
Oct 30 22:04:09.371587 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.371445    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-workload-socket\") pod \"727b252d-ad3e-4e43-9fb8-471fd2957231\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") "
Oct 30 22:04:09.371587 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.371476    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-workload-certs\") pod \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\" (UID: \"146b54d0-dc9b-4d22-9c30-fe25bd1d5194\") "
Oct 30 22:04:09.371587 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.371505    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-istio-envoy\") pod \"727b252d-ad3e-4e43-9fb8-471fd2957231\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") "
Oct 30 22:04:09.371587 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.371549    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/727b252d-ad3e-4e43-9fb8-471fd2957231-istio-token\" (UniqueName: \"kubernetes.io/projected/727b252d-ad3e-4e43-9fb8-471fd2957231-istio-token\") pod \"727b252d-ad3e-4e43-9fb8-471fd2957231\" (UID: \"727b252d-ad3e-4e43-9fb8-471fd2957231\") "
Oct 30 22:04:09.371778 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.371701    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-credential-socket" pod "146b54d0-dc9b-4d22-9c30-fe25bd1d5194" (UID: "146b54d0-dc9b-4d22-9c30-fe25bd1d5194"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:09.371912 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.371892    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-workload-certs" pod "727b252d-ad3e-4e43-9fb8-471fd2957231" (UID: "727b252d-ad3e-4e43-9fb8-471fd2957231"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:09.372338 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.372153    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/727b252d-ad3e-4e43-9fb8-471fd2957231-istiod-ca-cert" pod "727b252d-ad3e-4e43-9fb8-471fd2957231" (UID: "727b252d-ad3e-4e43-9fb8-471fd2957231"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:09.372338 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.372316    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.372562 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.372340    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n4xw\" (UniqueName: \"kubernetes.io/projected/894d8554-651d-4ad9-8521-da6c98edca6d-kube-api-access-2n4xw\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.372562 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.372384    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.372562 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.372397    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.372562 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.372411    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/894d8554-651d-4ad9-8521-da6c98edca6d-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.372562 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.372424    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.372562 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.372437    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/894d8554-651d-4ad9-8521-da6c98edca6d-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.372562 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.372453    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/727b252d-ad3e-4e43-9fb8-471fd2957231-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.372562 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.372468    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.372562 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.372482    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.372562 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.372495    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/894d8554-651d-4ad9-8521-da6c98edca6d-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.372562 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.372509    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.372562 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.372523    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/894d8554-651d-4ad9-8521-da6c98edca6d-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.375317 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.373206    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-credential-socket" pod "727b252d-ad3e-4e43-9fb8-471fd2957231" (UID: "727b252d-ad3e-4e43-9fb8-471fd2957231"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:09.375317 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.373434    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-istio-data" pod "727b252d-ad3e-4e43-9fb8-471fd2957231" (UID: "727b252d-ad3e-4e43-9fb8-471fd2957231"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:09.375317 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.374422    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-workload-socket" pod "727b252d-ad3e-4e43-9fb8-471fd2957231" (UID: "727b252d-ad3e-4e43-9fb8-471fd2957231"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:09.375317 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.374812    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istiod-ca-cert" pod "146b54d0-dc9b-4d22-9c30-fe25bd1d5194" (UID: "146b54d0-dc9b-4d22-9c30-fe25bd1d5194"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:09.375979 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.375764    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istio-data" pod "146b54d0-dc9b-4d22-9c30-fe25bd1d5194" (UID: "146b54d0-dc9b-4d22-9c30-fe25bd1d5194"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:09.375979 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.375813    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-workload-certs" pod "146b54d0-dc9b-4d22-9c30-fe25bd1d5194" (UID: "146b54d0-dc9b-4d22-9c30-fe25bd1d5194"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:09.383849 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.383781    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-istio-envoy" pod "727b252d-ad3e-4e43-9fb8-471fd2957231" (UID: "727b252d-ad3e-4e43-9fb8-471fd2957231"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:09.384334 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.384305    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istio-token" pod "146b54d0-dc9b-4d22-9c30-fe25bd1d5194" (UID: "146b54d0-dc9b-4d22-9c30-fe25bd1d5194"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:09.384548 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.384521    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-kube-api-access-tz6bj" pod "146b54d0-dc9b-4d22-9c30-fe25bd1d5194" (UID: "146b54d0-dc9b-4d22-9c30-fe25bd1d5194"). InnerVolumeSpecName "kube-api-access-tz6bj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:09.384982 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.384950    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/727b252d-ad3e-4e43-9fb8-471fd2957231-kube-api-access-dfk2t" pod "727b252d-ad3e-4e43-9fb8-471fd2957231" (UID: "727b252d-ad3e-4e43-9fb8-471fd2957231"). InnerVolumeSpecName "kube-api-access-dfk2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:09.385520 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.385471    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istio-podinfo" pod "146b54d0-dc9b-4d22-9c30-fe25bd1d5194" (UID: "146b54d0-dc9b-4d22-9c30-fe25bd1d5194"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:04:09.385807 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.385776    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/727b252d-ad3e-4e43-9fb8-471fd2957231-istio-podinfo" pod "727b252d-ad3e-4e43-9fb8-471fd2957231" (UID: "727b252d-ad3e-4e43-9fb8-471fd2957231"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:04:09.386078 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.386043    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/727b252d-ad3e-4e43-9fb8-471fd2957231-istio-token" pod "727b252d-ad3e-4e43-9fb8-471fd2957231" (UID: "727b252d-ad3e-4e43-9fb8-471fd2957231"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:09.386248 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.386221    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istio-envoy" pod "146b54d0-dc9b-4d22-9c30-fe25bd1d5194" (UID: "146b54d0-dc9b-4d22-9c30-fe25bd1d5194"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:09.474418 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.474365    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.474418 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.474408    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.474679 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.474425    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfk2t\" (UniqueName: \"kubernetes.io/projected/727b252d-ad3e-4e43-9fb8-471fd2957231-kube-api-access-dfk2t\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.474679 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.474440    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.474679 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.474454    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.474679 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.474467    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.474679 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.474480    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.474679 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.474493    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.474679 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.474506    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/727b252d-ad3e-4e43-9fb8-471fd2957231-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.474679 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.474519    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/727b252d-ad3e-4e43-9fb8-471fd2957231-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.474679 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.474532    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz6bj\" (UniqueName: \"kubernetes.io/projected/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-kube-api-access-tz6bj\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.474679 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.474545    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.474679 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.474563    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/727b252d-ad3e-4e43-9fb8-471fd2957231-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.474679 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.474576    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/146b54d0-dc9b-4d22-9c30-fe25bd1d5194-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:09.904208 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.904125    5087 generic.go:356] "Generic (PLEG): container finished" podID="aed464f0-0b79-467a-b131-fa6944d01941" containerID="a82229b4acff245f57189518bbc0d179e406e82a54c2debb702dff2a9233a7e5" exitCode=0
Oct 30 22:04:09.904909 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.904475    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp" event={"ID":"aed464f0-0b79-467a-b131-fa6944d01941","Type":"ContainerDied","Data":"a82229b4acff245f57189518bbc0d179e406e82a54c2debb702dff2a9233a7e5"}
Oct 30 22:04:09.909920 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.909166    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg" event={"ID":"894d8554-651d-4ad9-8521-da6c98edca6d","Type":"ContainerDied","Data":"45a37f156b8f8e3b554bb6e2436998440de00e4db128b13d0dd65b40dedce169"}
Oct 30 22:04:09.909920 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.909233    5087 scope.go:122] "RemoveContainer" containerID="44faaa6dc01eadd1454099f86f68e37eb2d12eae44ef8d995c62974668560deb"
Oct 30 22:04:09.909920 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.909406    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"
Oct 30 22:04:09.914150 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.913540    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7" event={"ID":"146b54d0-dc9b-4d22-9c30-fe25bd1d5194","Type":"ContainerDied","Data":"19e5311fb39a65431f1ad0523aaffb1e7a4310b54367fbe1498f7e90100cdf18"}
Oct 30 22:04:09.914150 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.913643    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"
Oct 30 22:04:09.917671 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.917423    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k" event={"ID":"89d4443a-0fb1-461e-a866-05351817130e","Type":"ContainerDied","Data":"78942715bc4cbc7489bf644a09e1a74f99ccc5cc4d58cbd4f841fd0ec61790bc"}
Oct 30 22:04:09.917671 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.917537    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"
Oct 30 22:04:09.921058 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.920773    5087 generic.go:356] "Generic (PLEG): container finished" podID="727b252d-ad3e-4e43-9fb8-471fd2957231" containerID="2a184e22927900b74b6226bf17a18ebf3d0c92e1216ff49aad0d86bc8c54e247" exitCode=0
Oct 30 22:04:09.921058 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.920866    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96" event={"ID":"727b252d-ad3e-4e43-9fb8-471fd2957231","Type":"ContainerDied","Data":"2a184e22927900b74b6226bf17a18ebf3d0c92e1216ff49aad0d86bc8c54e247"}
Oct 30 22:04:09.921058 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.920899    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96" event={"ID":"727b252d-ad3e-4e43-9fb8-471fd2957231","Type":"ContainerDied","Data":"79367c6f291c911af631d88b7ea6c6f99c0bf9c0aa7b427374b528248873d2f4"}
Oct 30 22:04:09.921058 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.920955    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"
Oct 30 22:04:09.933180 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.933137    5087 scope.go:122] "RemoveContainer" containerID="92fff8afe405ce49e361b133b95ed80dd7e4ab927054e393d22924b42f9ea685"
Oct 30 22:04:09.947480 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.947457    5087 scope.go:122] "RemoveContainer" containerID="a995abf6d93c7e70bb189419720531fb8bf63b9983121430844125d99718b834"
Oct 30 22:04:09.995160 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.995137    5087 scope.go:122] "RemoveContainer" containerID="2a184e22927900b74b6226bf17a18ebf3d0c92e1216ff49aad0d86bc8c54e247"
Oct 30 22:04:09.997136 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:09.997110    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 22:04:10.005247 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.005231    5087 scope.go:122] "RemoveContainer" containerID="2a184e22927900b74b6226bf17a18ebf3d0c92e1216ff49aad0d86bc8c54e247"
Oct 30 22:04:10.005566 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:04:10.005547    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a184e22927900b74b6226bf17a18ebf3d0c92e1216ff49aad0d86bc8c54e247\": container with ID starting with 2a184e22927900b74b6226bf17a18ebf3d0c92e1216ff49aad0d86bc8c54e247 not found: ID does not exist" containerID="2a184e22927900b74b6226bf17a18ebf3d0c92e1216ff49aad0d86bc8c54e247"
Oct 30 22:04:10.005638 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.005574    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a184e22927900b74b6226bf17a18ebf3d0c92e1216ff49aad0d86bc8c54e247"} err="failed to get container status \"2a184e22927900b74b6226bf17a18ebf3d0c92e1216ff49aad0d86bc8c54e247\": rpc error: code = NotFound desc = could not find container \"2a184e22927900b74b6226bf17a18ebf3d0c92e1216ff49aad0d86bc8c54e247\": container with ID starting with 2a184e22927900b74b6226bf17a18ebf3d0c92e1216ff49aad0d86bc8c54e247 not found: ID does not exist"
Oct 30 22:04:10.077933 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.077892    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"]
Oct 30 22:04:10.081636 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.081611    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-l6mk7"]
Oct 30 22:04:10.093086 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.093060    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-istio-data\") pod \"aed464f0-0b79-467a-b131-fa6944d01941\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") "
Oct 30 22:04:10.093510 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.093173    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-workload-socket\") pod \"aed464f0-0b79-467a-b131-fa6944d01941\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") "
Oct 30 22:04:10.093510 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.093227    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aed464f0-0b79-467a-b131-fa6944d01941-kube-api-access-rvncm\" (UniqueName: \"kubernetes.io/projected/aed464f0-0b79-467a-b131-fa6944d01941-kube-api-access-rvncm\") pod \"aed464f0-0b79-467a-b131-fa6944d01941\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") "
Oct 30 22:04:10.093510 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.093280    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-istio-envoy\") pod \"aed464f0-0b79-467a-b131-fa6944d01941\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") "
Oct 30 22:04:10.093510 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.093335    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-credential-socket\") pod \"aed464f0-0b79-467a-b131-fa6944d01941\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") "
Oct 30 22:04:10.093510 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.093443    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aed464f0-0b79-467a-b131-fa6944d01941-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed464f0-0b79-467a-b131-fa6944d01941-istiod-ca-cert\") pod \"aed464f0-0b79-467a-b131-fa6944d01941\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") "
Oct 30 22:04:10.093510 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.093508    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-workload-certs\") pod \"aed464f0-0b79-467a-b131-fa6944d01941\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") "
Oct 30 22:04:10.094100 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.093676    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aed464f0-0b79-467a-b131-fa6944d01941-istio-token\" (UniqueName: \"kubernetes.io/projected/aed464f0-0b79-467a-b131-fa6944d01941-istio-token\") pod \"aed464f0-0b79-467a-b131-fa6944d01941\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") "
Oct 30 22:04:10.094100 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.093747    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aed464f0-0b79-467a-b131-fa6944d01941-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed464f0-0b79-467a-b131-fa6944d01941-istio-podinfo\") pod \"aed464f0-0b79-467a-b131-fa6944d01941\" (UID: \"aed464f0-0b79-467a-b131-fa6944d01941\") "
Oct 30 22:04:10.094100 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.093779    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-workload-socket" pod "aed464f0-0b79-467a-b131-fa6944d01941" (UID: "aed464f0-0b79-467a-b131-fa6944d01941"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:10.094100 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.094068    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-workload-certs" pod "aed464f0-0b79-467a-b131-fa6944d01941" (UID: "aed464f0-0b79-467a-b131-fa6944d01941"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:10.094358 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.094320    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aed464f0-0b79-467a-b131-fa6944d01941-istiod-ca-cert" pod "aed464f0-0b79-467a-b131-fa6944d01941" (UID: "aed464f0-0b79-467a-b131-fa6944d01941"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:10.094455 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.094423    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:10.094455 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.094443    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:10.094811 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.094781    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-istio-data" pod "aed464f0-0b79-467a-b131-fa6944d01941" (UID: "aed464f0-0b79-467a-b131-fa6944d01941"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:10.095246 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.094976    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-credential-socket" pod "aed464f0-0b79-467a-b131-fa6944d01941" (UID: "aed464f0-0b79-467a-b131-fa6944d01941"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:10.098624 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.098592    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed464f0-0b79-467a-b131-fa6944d01941-istio-token" pod "aed464f0-0b79-467a-b131-fa6944d01941" (UID: "aed464f0-0b79-467a-b131-fa6944d01941"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:10.098766 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.098706    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-istio-envoy" pod "aed464f0-0b79-467a-b131-fa6944d01941" (UID: "aed464f0-0b79-467a-b131-fa6944d01941"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:04:10.100835 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.100812    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aed464f0-0b79-467a-b131-fa6944d01941-istio-podinfo" pod "aed464f0-0b79-467a-b131-fa6944d01941" (UID: "aed464f0-0b79-467a-b131-fa6944d01941"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:04:10.102908 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.102881    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed464f0-0b79-467a-b131-fa6944d01941-kube-api-access-rvncm" pod "aed464f0-0b79-467a-b131-fa6944d01941" (UID: "aed464f0-0b79-467a-b131-fa6944d01941"). InnerVolumeSpecName "kube-api-access-rvncm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:10.192805 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.192712    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"]
Oct 30 22:04:10.195790 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.195759    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aed464f0-0b79-467a-b131-fa6944d01941-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:10.195790 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.195788    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aed464f0-0b79-467a-b131-fa6944d01941-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:10.196026 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.195799    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:10.196026 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.195809    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvncm\" (UniqueName: \"kubernetes.io/projected/aed464f0-0b79-467a-b131-fa6944d01941-kube-api-access-rvncm\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:10.196026 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.195821    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:10.196026 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.195835    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aed464f0-0b79-467a-b131-fa6944d01941-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:10.196026 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.195843    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aed464f0-0b79-467a-b131-fa6944d01941-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:04:10.197088 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.197062    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-4kk96"]
Oct 30 22:04:10.304681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.304649    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"]
Oct 30 22:04:10.311589 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.311567    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-dhs4k"]
Oct 30 22:04:10.421977 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.421947    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"]
Oct 30 22:04:10.425821 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.425794    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-80c366-openshift-default-666b7db455-6ztpg"]
Oct 30 22:04:10.934227 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.934187    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp" event={"ID":"aed464f0-0b79-467a-b131-fa6944d01941","Type":"ContainerDied","Data":"b73c2ca4ec3fadbd01d458b2a8e949a8e59cee8000bc4fa5277fccd20e677f2d"}
Oct 30 22:04:10.934693 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.934245    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"
Oct 30 22:04:10.934693 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:10.934265    5087 scope.go:122] "RemoveContainer" containerID="a82229b4acff245f57189518bbc0d179e406e82a54c2debb702dff2a9233a7e5"
Oct 30 22:04:11.111572 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:11.111532    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"]
Oct 30 22:04:11.119635 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:11.119606    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-ldkbp"]
Oct 30 22:04:11.711034 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:11.710992    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="146b54d0-dc9b-4d22-9c30-fe25bd1d5194" path="/var/lib/kubelet/pods/146b54d0-dc9b-4d22-9c30-fe25bd1d5194/volumes"
Oct 30 22:04:11.711564 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:11.711550    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="727b252d-ad3e-4e43-9fb8-471fd2957231" path="/var/lib/kubelet/pods/727b252d-ad3e-4e43-9fb8-471fd2957231/volumes"
Oct 30 22:04:11.711978 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:11.711966    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="894d8554-651d-4ad9-8521-da6c98edca6d" path="/var/lib/kubelet/pods/894d8554-651d-4ad9-8521-da6c98edca6d/volumes"
Oct 30 22:04:11.712774 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:11.712758    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89d4443a-0fb1-461e-a866-05351817130e" path="/var/lib/kubelet/pods/89d4443a-0fb1-461e-a866-05351817130e/volumes"
Oct 30 22:04:11.713176 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:11.713164    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aed464f0-0b79-467a-b131-fa6944d01941" path="/var/lib/kubelet/pods/aed464f0-0b79-467a-b131-fa6944d01941/volumes"
Oct 30 22:04:17.169063 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:04:17.169027    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:04:28.641809 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:28.641779    5087 scope.go:122] "RemoveContainer" containerID="bf14abaffed3d1421e2feb0d542c2e50f6ea49d211407418327b73fc64de896e"
Oct 30 22:04:28.652444 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:28.652418    5087 scope.go:122] "RemoveContainer" containerID="a52679231d012bc4dbb194c078a18b81ed99b5de68c9994f845b379bc01caf62"
Oct 30 22:04:28.662922 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:28.662902    5087 scope.go:122] "RemoveContainer" containerID="049e438fadb172173726c66e97e5da3bb67e3c3c60108ca22738eac1318c5509"
Oct 30 22:04:47.169578 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:04:47.169535    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:04:52.052137 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.052101    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e611abd0-76s48"]
Oct 30 22:04:52.103566 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.103526    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e611abd0-76s48"]
Oct 30 22:04:52.106053 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.106027    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-485d1e-58bccc79c9-jk9hv"]
Oct 30 22:04:52.106771 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.106752    5087 cpu_manager.go:414] "containerMap: removing container" podUID="5337230d-c32c-42e6-8985-4ff157dd5b95" containerName="istio-proxy"
Oct 30 22:04:52.106771 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.106771    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5337230d-c32c-42e6-8985-4ff157dd5b95" containerName="istio-proxy"
Oct 30 22:04:52.106939 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.106793    5087 cpu_manager.go:414] "containerMap: removing container" podUID="aed464f0-0b79-467a-b131-fa6944d01941" containerName="istio-proxy"
Oct 30 22:04:52.106939 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.106798    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aed464f0-0b79-467a-b131-fa6944d01941" containerName="istio-proxy"
Oct 30 22:04:52.106939 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.106842    5087 cpu_manager.go:414] "containerMap: removing container" podUID="727b252d-ad3e-4e43-9fb8-471fd2957231" containerName="istio-proxy"
Oct 30 22:04:52.106939 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.106847    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="727b252d-ad3e-4e43-9fb8-471fd2957231" containerName="istio-proxy"
Oct 30 22:04:52.106939 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.106856    5087 cpu_manager.go:414] "containerMap: removing container" podUID="09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b" containerName="istio-proxy"
Oct 30 22:04:52.106939 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.106862    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b" containerName="istio-proxy"
Oct 30 22:04:52.106939 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.106883    5087 cpu_manager.go:414] "containerMap: removing container" podUID="894d8554-651d-4ad9-8521-da6c98edca6d" containerName="istio-proxy"
Oct 30 22:04:52.106939 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.106890    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="894d8554-651d-4ad9-8521-da6c98edca6d" containerName="istio-proxy"
Oct 30 22:04:52.106939 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.106909    5087 cpu_manager.go:414] "containerMap: removing container" podUID="89d4443a-0fb1-461e-a866-05351817130e" containerName="istio-proxy"
Oct 30 22:04:52.106939 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.106917    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89d4443a-0fb1-461e-a866-05351817130e" containerName="istio-proxy"
Oct 30 22:04:52.106939 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.106938    5087 cpu_manager.go:414] "containerMap: removing container" podUID="b0bffa69-f8de-4190-a9b9-88b9fa0ff17b" containerName="revoke-keys"
Oct 30 22:04:52.106939 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.106945    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0bffa69-f8de-4190-a9b9-88b9fa0ff17b" containerName="revoke-keys"
Oct 30 22:04:52.107568 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.106969    5087 cpu_manager.go:414] "containerMap: removing container" podUID="3f00bbc1-19c4-447a-9459-3443bbabb845" containerName="istio-proxy"
Oct 30 22:04:52.107568 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.106975    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f00bbc1-19c4-447a-9459-3443bbabb845" containerName="istio-proxy"
Oct 30 22:04:52.107568 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.106987    5087 cpu_manager.go:414] "containerMap: removing container" podUID="146b54d0-dc9b-4d22-9c30-fe25bd1d5194" containerName="istio-proxy"
Oct 30 22:04:52.107568 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.106993    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="146b54d0-dc9b-4d22-9c30-fe25bd1d5194" containerName="istio-proxy"
Oct 30 22:04:52.107568 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.107159    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="aed464f0-0b79-467a-b131-fa6944d01941" containerName="istio-proxy"
Oct 30 22:04:52.107568 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.107177    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="727b252d-ad3e-4e43-9fb8-471fd2957231" containerName="istio-proxy"
Oct 30 22:04:52.107568 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.107183    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="09a8a5cd-2a5e-4afb-9aae-7680fedb0c2b" containerName="istio-proxy"
Oct 30 22:04:52.107568 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.107191    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="894d8554-651d-4ad9-8521-da6c98edca6d" containerName="istio-proxy"
Oct 30 22:04:52.107568 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.107196    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="89d4443a-0fb1-461e-a866-05351817130e" containerName="istio-proxy"
Oct 30 22:04:52.107568 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.107203    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0bffa69-f8de-4190-a9b9-88b9fa0ff17b" containerName="revoke-keys"
Oct 30 22:04:52.107568 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.107211    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f00bbc1-19c4-447a-9459-3443bbabb845" containerName="istio-proxy"
Oct 30 22:04:52.107568 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.107216    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="146b54d0-dc9b-4d22-9c30-fe25bd1d5194" containerName="istio-proxy"
Oct 30 22:04:52.107568 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.107224    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="5337230d-c32c-42e6-8985-4ff157dd5b95" containerName="istio-proxy"
Oct 30 22:04:52.111139 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.111119    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-485d1e-58bccc79c9-jk9hv"
Oct 30 22:04:52.163620 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.163586    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-485d1e\""
Oct 30 22:04:52.163837 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.163728    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-485d1e-dockercfg-r98kl\""
Oct 30 22:04:52.167983 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.167944    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-485d1e-58bccc79c9-jk9hv"]
Oct 30 22:04:52.223483 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.223445    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chlq8\" (UniqueName: \"kubernetes.io/projected/94fcd256-faba-490e-9680-d986a9635413-kube-api-access-chlq8\") pod \"payload-pre-processing-e2e-shared-b-w0-485d1e-58bccc79c9-jk9hv\" (UID: \"94fcd256-faba-490e-9680-d986a9635413\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-485d1e-58bccc79c9-jk9hv"
Oct 30 22:04:52.223720 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.223699    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94fcd256-faba-490e-9680-d986a9635413-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-485d1e-58bccc79c9-jk9hv\" (UID: \"94fcd256-faba-490e-9680-d986a9635413\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-485d1e-58bccc79c9-jk9hv"
Oct 30 22:04:52.332415 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.323074    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-485d1e-66c964c857-fddzn"]
Oct 30 22:04:52.332415 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.327561    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chlq8\" (UniqueName: \"kubernetes.io/projected/94fcd256-faba-490e-9680-d986a9635413-kube-api-access-chlq8\") pod \"payload-pre-processing-e2e-shared-b-w0-485d1e-58bccc79c9-jk9hv\" (UID: \"94fcd256-faba-490e-9680-d986a9635413\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-485d1e-58bccc79c9-jk9hv"
Oct 30 22:04:52.332415 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.327808    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94fcd256-faba-490e-9680-d986a9635413-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-485d1e-58bccc79c9-jk9hv\" (UID: \"94fcd256-faba-490e-9680-d986a9635413\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-485d1e-58bccc79c9-jk9hv"
Oct 30 22:04:52.332821 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.332782    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94fcd256-faba-490e-9680-d986a9635413-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-485d1e-58bccc79c9-jk9hv\" (UID: \"94fcd256-faba-490e-9680-d986a9635413\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-485d1e-58bccc79c9-jk9hv"
Oct 30 22:04:52.346995 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.345445    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-485d1e-66c964c857-fddzn"
Oct 30 22:04:52.387310 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.384772    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-485d1e-66c964c857-fddzn"]
Oct 30 22:04:52.429923 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.429887    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72563fad-9b54-4005-a4eb-1466011220f8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-485d1e-66c964c857-fddzn\" (UID: \"72563fad-9b54-4005-a4eb-1466011220f8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-485d1e-66c964c857-fddzn"
Oct 30 22:04:52.430187 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.430164    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66tv6\" (UniqueName: \"kubernetes.io/projected/72563fad-9b54-4005-a4eb-1466011220f8-kube-api-access-66tv6\") pod \"payload-processing-e2e-shared-b-w0-485d1e-66c964c857-fddzn\" (UID: \"72563fad-9b54-4005-a4eb-1466011220f8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-485d1e-66c964c857-fddzn"
Oct 30 22:04:52.431321 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.431300    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chlq8\" (UniqueName: \"kubernetes.io/projected/94fcd256-faba-490e-9680-d986a9635413-kube-api-access-chlq8\") pod \"payload-pre-processing-e2e-shared-b-w0-485d1e-58bccc79c9-jk9hv\" (UID: \"94fcd256-faba-490e-9680-d986a9635413\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-485d1e-58bccc79c9-jk9hv"
Oct 30 22:04:52.531323 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.531291    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72563fad-9b54-4005-a4eb-1466011220f8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-485d1e-66c964c857-fddzn\" (UID: \"72563fad-9b54-4005-a4eb-1466011220f8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-485d1e-66c964c857-fddzn"
Oct 30 22:04:52.531542 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.531468    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66tv6\" (UniqueName: \"kubernetes.io/projected/72563fad-9b54-4005-a4eb-1466011220f8-kube-api-access-66tv6\") pod \"payload-processing-e2e-shared-b-w0-485d1e-66c964c857-fddzn\" (UID: \"72563fad-9b54-4005-a4eb-1466011220f8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-485d1e-66c964c857-fddzn"
Oct 30 22:04:52.532121 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.532101    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72563fad-9b54-4005-a4eb-1466011220f8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-485d1e-66c964c857-fddzn\" (UID: \"72563fad-9b54-4005-a4eb-1466011220f8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-485d1e-66c964c857-fddzn"
Oct 30 22:04:52.589365 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.589244    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66tv6\" (UniqueName: \"kubernetes.io/projected/72563fad-9b54-4005-a4eb-1466011220f8-kube-api-access-66tv6\") pod \"payload-processing-e2e-shared-b-w0-485d1e-66c964c857-fddzn\" (UID: \"72563fad-9b54-4005-a4eb-1466011220f8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-485d1e-66c964c857-fddzn"
Oct 30 22:04:52.694446 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.694408    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-485d1e-66c964c857-fddzn"
Oct 30 22:04:52.726739 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:52.726710    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-485d1e-58bccc79c9-jk9hv"
Oct 30 22:04:53.083116 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:53.083087    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-485d1e-66c964c857-fddzn"]
Oct 30 22:04:53.084538 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:04:53.084505    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72563fad_9b54_4005_a4eb_1466011220f8.slice/crio-f8ec9af6c1c4f1014e24757cbdf34273634c3e8c45fe5ee599b7d1f421f38db8 WatchSource:0}: Error finding container f8ec9af6c1c4f1014e24757cbdf34273634c3e8c45fe5ee599b7d1f421f38db8: Status 404 returned error can't find the container with id f8ec9af6c1c4f1014e24757cbdf34273634c3e8c45fe5ee599b7d1f421f38db8
Oct 30 22:04:53.147660 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:53.147587    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-485d1e-58bccc79c9-jk9hv"]
Oct 30 22:04:53.150202 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:04:53.150172    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94fcd256_faba_490e_9680_d986a9635413.slice/crio-81a2ce5cc2e7b89074d161c00625571e694104f608800c18055abac89c17c373 WatchSource:0}: Error finding container 81a2ce5cc2e7b89074d161c00625571e694104f608800c18055abac89c17c373: Status 404 returned error can't find the container with id 81a2ce5cc2e7b89074d161c00625571e694104f608800c18055abac89c17c373
Oct 30 22:04:53.357313 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:53.357216    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-485d1e-66c964c857-fddzn" event={"ID":"72563fad-9b54-4005-a4eb-1466011220f8","Type":"ContainerStarted","Data":"8c0e53a81e7fe393befbe0df60af66331c8288d0eca4b93deb949306d4da6b2d"}
Oct 30 22:04:53.357313 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:53.357282    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-485d1e-66c964c857-fddzn" event={"ID":"72563fad-9b54-4005-a4eb-1466011220f8","Type":"ContainerStarted","Data":"f8ec9af6c1c4f1014e24757cbdf34273634c3e8c45fe5ee599b7d1f421f38db8"}
Oct 30 22:04:53.357566 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:53.357465    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-485d1e-66c964c857-fddzn"
Oct 30 22:04:53.359164 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:53.359133    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-485d1e-58bccc79c9-jk9hv" event={"ID":"94fcd256-faba-490e-9680-d986a9635413","Type":"ContainerStarted","Data":"b156f7d992db23f26da07c58f46ab9dcf72b827dff754a7b06614db837c7a684"}
Oct 30 22:04:53.359324 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:53.359184    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-485d1e-58bccc79c9-jk9hv" event={"ID":"94fcd256-faba-490e-9680-d986a9635413","Type":"ContainerStarted","Data":"81a2ce5cc2e7b89074d161c00625571e694104f608800c18055abac89c17c373"}
Oct 30 22:04:53.359324 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:53.359236    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-485d1e-58bccc79c9-jk9hv"
Oct 30 22:04:53.474918 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:53.474862    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-485d1e-66c964c857-fddzn" podStartSLOduration=1.474844498 podStartE2EDuration="1.474844498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:53.472297854 +0000 UTC m=+1887.946801834" watchObservedRunningTime="2026-10-30 22:04:53.474844498 +0000 UTC m=+1887.949348478"
Oct 30 22:04:53.598909 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:53.597358    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-485d1e-58bccc79c9-jk9hv" podStartSLOduration=1.597319083 podStartE2EDuration="1.597319083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:53.593606593 +0000 UTC m=+1888.068110575" watchObservedRunningTime="2026-10-30 22:04:53.597319083 +0000 UTC m=+1888.071823063"
Oct 30 22:04:53.712120 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:04:53.712027    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b73395fa-cf8c-42d3-be5b-1f12e224d56f" path="/var/lib/kubelet/pods/b73395fa-cf8c-42d3-be5b-1f12e224d56f/volumes"
Oct 30 22:05:04.369524 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:04.369490    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-485d1e-58bccc79c9-jk9hv"
Oct 30 22:05:04.370576 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:04.369600    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-485d1e-66c964c857-fddzn"
Oct 30 22:05:12.948413 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:12.948364    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk"]
Oct 30 22:05:12.953991 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:12.953970    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk"
Oct 30 22:05:13.011484 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:13.011456    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk"]
Oct 30 22:05:13.040701 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:13.040668    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8m58\" (UniqueName: \"kubernetes.io/projected/294e4f05-6ceb-4441-b6dd-e9e197048ea6-kube-api-access-m8m58\") pod \"maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk\" (UID: \"294e4f05-6ceb-4441-b6dd-e9e197048ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk"
Oct 30 22:05:13.040892 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:13.040826    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294e4f05-6ceb-4441-b6dd-e9e197048ea6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk\" (UID: \"294e4f05-6ceb-4441-b6dd-e9e197048ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk"
Oct 30 22:05:13.041095 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:13.041072    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294e4f05-6ceb-4441-b6dd-e9e197048ea6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk\" (UID: \"294e4f05-6ceb-4441-b6dd-e9e197048ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk"
Oct 30 22:05:13.056334 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:13.056302    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:05:13.056498 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:13.056412    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:05:13.142879 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:13.142833    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294e4f05-6ceb-4441-b6dd-e9e197048ea6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk\" (UID: \"294e4f05-6ceb-4441-b6dd-e9e197048ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk"
Oct 30 22:05:13.143102 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:13.143081    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294e4f05-6ceb-4441-b6dd-e9e197048ea6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk\" (UID: \"294e4f05-6ceb-4441-b6dd-e9e197048ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk"
Oct 30 22:05:13.143223 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:13.143206    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8m58\" (UniqueName: \"kubernetes.io/projected/294e4f05-6ceb-4441-b6dd-e9e197048ea6-kube-api-access-m8m58\") pod \"maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk\" (UID: \"294e4f05-6ceb-4441-b6dd-e9e197048ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk"
Oct 30 22:05:13.143895 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:13.143865    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294e4f05-6ceb-4441-b6dd-e9e197048ea6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk\" (UID: \"294e4f05-6ceb-4441-b6dd-e9e197048ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk"
Oct 30 22:05:13.145460 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:13.145439    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294e4f05-6ceb-4441-b6dd-e9e197048ea6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk\" (UID: \"294e4f05-6ceb-4441-b6dd-e9e197048ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk"
Oct 30 22:05:13.200293 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:13.200225    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8m58\" (UniqueName: \"kubernetes.io/projected/294e4f05-6ceb-4441-b6dd-e9e197048ea6-kube-api-access-m8m58\") pod \"maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk\" (UID: \"294e4f05-6ceb-4441-b6dd-e9e197048ea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk"
Oct 30 22:05:13.273552 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:13.273516    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk"
Oct 30 22:05:13.656742 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:13.656711    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk"]
Oct 30 22:05:13.657385 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:05:13.657358    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod294e4f05_6ceb_4441_b6dd_e9e197048ea6.slice/crio-4afbf0fe14a03c823aa97a7b4bcc7b14f4dd6d4cc02d47351846518d2294667c WatchSource:0}: Error finding container 4afbf0fe14a03c823aa97a7b4bcc7b14f4dd6d4cc02d47351846518d2294667c: Status 404 returned error can't find the container with id 4afbf0fe14a03c823aa97a7b4bcc7b14f4dd6d4cc02d47351846518d2294667c
Oct 30 22:05:14.533628 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:14.533589    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk" event={"ID":"294e4f05-6ceb-4441-b6dd-e9e197048ea6","Type":"ContainerStarted","Data":"8c1484516cc037d23ee71b072f8dde098e9058bbe996ba0236d6b7f1b9a51d34"}
Oct 30 22:05:14.534035 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:14.533678    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk" event={"ID":"294e4f05-6ceb-4441-b6dd-e9e197048ea6","Type":"ContainerStarted","Data":"4afbf0fe14a03c823aa97a7b4bcc7b14f4dd6d4cc02d47351846518d2294667c"}
Oct 30 22:05:14.651155 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:14.651103    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk" podStartSLOduration=2.651090099 podStartE2EDuration="2.651090099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:14.647728597 +0000 UTC m=+1909.122232575" watchObservedRunningTime="2026-10-30 22:05:14.651090099 +0000 UTC m=+1909.125594072"
Oct 30 22:05:15.544688 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:15.544650    5087 generic.go:356] "Generic (PLEG): container finished" podID="294e4f05-6ceb-4441-b6dd-e9e197048ea6" containerID="8c1484516cc037d23ee71b072f8dde098e9058bbe996ba0236d6b7f1b9a51d34" exitCode=0
Oct 30 22:05:15.545235 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:15.544762    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk" event={"ID":"294e4f05-6ceb-4441-b6dd-e9e197048ea6","Type":"ContainerDied","Data":"8c1484516cc037d23ee71b072f8dde098e9058bbe996ba0236d6b7f1b9a51d34"}
Oct 30 22:05:16.722217 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:16.722189    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk"
Oct 30 22:05:16.805309 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:16.805231    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/294e4f05-6ceb-4441-b6dd-e9e197048ea6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294e4f05-6ceb-4441-b6dd-e9e197048ea6-maas-api-openshift-ca\") pod \"294e4f05-6ceb-4441-b6dd-e9e197048ea6\" (UID: \"294e4f05-6ceb-4441-b6dd-e9e197048ea6\") "
Oct 30 22:05:16.805564 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:16.805542    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/294e4f05-6ceb-4441-b6dd-e9e197048ea6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294e4f05-6ceb-4441-b6dd-e9e197048ea6-maas-api-xks-ca\") pod \"294e4f05-6ceb-4441-b6dd-e9e197048ea6\" (UID: \"294e4f05-6ceb-4441-b6dd-e9e197048ea6\") "
Oct 30 22:05:16.805675 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:16.805659    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/294e4f05-6ceb-4441-b6dd-e9e197048ea6-kube-api-access-m8m58\" (UniqueName: \"kubernetes.io/projected/294e4f05-6ceb-4441-b6dd-e9e197048ea6-kube-api-access-m8m58\") pod \"294e4f05-6ceb-4441-b6dd-e9e197048ea6\" (UID: \"294e4f05-6ceb-4441-b6dd-e9e197048ea6\") "
Oct 30 22:05:16.805675 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:16.805663    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/294e4f05-6ceb-4441-b6dd-e9e197048ea6-maas-api-openshift-ca" pod "294e4f05-6ceb-4441-b6dd-e9e197048ea6" (UID: "294e4f05-6ceb-4441-b6dd-e9e197048ea6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:16.806380 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:16.806273    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/294e4f05-6ceb-4441-b6dd-e9e197048ea6-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:16.807816 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:16.807788    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/294e4f05-6ceb-4441-b6dd-e9e197048ea6-maas-api-xks-ca" pod "294e4f05-6ceb-4441-b6dd-e9e197048ea6" (UID: "294e4f05-6ceb-4441-b6dd-e9e197048ea6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:05:16.807926 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:16.807903    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/294e4f05-6ceb-4441-b6dd-e9e197048ea6-kube-api-access-m8m58" pod "294e4f05-6ceb-4441-b6dd-e9e197048ea6" (UID: "294e4f05-6ceb-4441-b6dd-e9e197048ea6"). InnerVolumeSpecName "kube-api-access-m8m58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:16.907941 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:16.907906    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/294e4f05-6ceb-4441-b6dd-e9e197048ea6-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:16.907941 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:16.907938    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8m58\" (UniqueName: \"kubernetes.io/projected/294e4f05-6ceb-4441-b6dd-e9e197048ea6-kube-api-access-m8m58\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:17.169318 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:05:17.169226    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:05:17.563542 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:17.563513    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk"
Oct 30 22:05:17.563740 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:17.563552    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk" event={"ID":"294e4f05-6ceb-4441-b6dd-e9e197048ea6","Type":"ContainerDied","Data":"4afbf0fe14a03c823aa97a7b4bcc7b14f4dd6d4cc02d47351846518d2294667c"}
Oct 30 22:05:17.563740 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:17.563603    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4afbf0fe14a03c823aa97a7b4bcc7b14f4dd6d4cc02d47351846518d2294667c"
Oct 30 22:05:28.724760 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:28.724732    5087 scope.go:122] "RemoveContainer" containerID="a2bb68712d151a05a5718090c69ea48179fefe49e51f449c14865ff61b58e52d"
Oct 30 22:05:38.054536 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:38.054454    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caaa84af-7s79k"]
Oct 30 22:05:38.103622 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:38.103593    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caaa84af-7s79k"]
Oct 30 22:05:39.711536 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:39.711446    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60216f24-7193-4fd9-86a5-92a1a9dd8f92" path="/var/lib/kubelet/pods/60216f24-7193-4fd9-86a5-92a1a9dd8f92/volumes"
Oct 30 22:05:47.296720 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:05:47.294519    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:05:50.156879 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:50.156844    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"]
Oct 30 22:05:50.157304 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:50.157133    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp" podUID="1b411531-02da-4593-a357-1a15f782c76b" containerName="istio-proxy" containerID="cri-o://21d90f075be8baa5083465e6c5d42b5d825eb96fc2a8d69ca0fe0f99bf5f6ab7" gracePeriod=30
Oct 30 22:05:53.798529 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:53.798492    5087 patch_prober.go:29] interesting pod/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:05:53.798918 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:53.798580    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp" podUID="1b411531-02da-4593-a357-1a15f782c76b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:05:55.770619 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.770588    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 22:05:55.863132 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.863101    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-workload-socket\") pod \"1b411531-02da-4593-a357-1a15f782c76b\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") "
Oct 30 22:05:55.863314 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.863145    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b411531-02da-4593-a357-1a15f782c76b-istio-token\" (UniqueName: \"kubernetes.io/projected/1b411531-02da-4593-a357-1a15f782c76b-istio-token\") pod \"1b411531-02da-4593-a357-1a15f782c76b\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") "
Oct 30 22:05:55.863314 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.863179    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b411531-02da-4593-a357-1a15f782c76b-kube-api-access-gxvfx\" (UniqueName: \"kubernetes.io/projected/1b411531-02da-4593-a357-1a15f782c76b-kube-api-access-gxvfx\") pod \"1b411531-02da-4593-a357-1a15f782c76b\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") "
Oct 30 22:05:55.863465 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.863375    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b411531-02da-4593-a357-1a15f782c76b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b411531-02da-4593-a357-1a15f782c76b-istiod-ca-cert\") pod \"1b411531-02da-4593-a357-1a15f782c76b\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") "
Oct 30 22:05:55.863465 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.863421    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-credential-socket\") pod \"1b411531-02da-4593-a357-1a15f782c76b\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") "
Oct 30 22:05:55.863465 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.863456    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-istio-envoy\") pod \"1b411531-02da-4593-a357-1a15f782c76b\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") "
Oct 30 22:05:55.863621 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.863458    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-workload-socket" pod "1b411531-02da-4593-a357-1a15f782c76b" (UID: "1b411531-02da-4593-a357-1a15f782c76b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:55.863621 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.863496    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-istio-data\") pod \"1b411531-02da-4593-a357-1a15f782c76b\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") "
Oct 30 22:05:55.863621 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.863584    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b411531-02da-4593-a357-1a15f782c76b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b411531-02da-4593-a357-1a15f782c76b-istio-podinfo\") pod \"1b411531-02da-4593-a357-1a15f782c76b\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") "
Oct 30 22:05:55.863769 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.863684    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-workload-certs\") pod \"1b411531-02da-4593-a357-1a15f782c76b\" (UID: \"1b411531-02da-4593-a357-1a15f782c76b\") "
Oct 30 22:05:55.863769 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.863717    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-credential-socket" pod "1b411531-02da-4593-a357-1a15f782c76b" (UID: "1b411531-02da-4593-a357-1a15f782c76b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:55.863866 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.863801    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b411531-02da-4593-a357-1a15f782c76b-istiod-ca-cert" pod "1b411531-02da-4593-a357-1a15f782c76b" (UID: "1b411531-02da-4593-a357-1a15f782c76b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:55.863944 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.863918    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-istio-data" pod "1b411531-02da-4593-a357-1a15f782c76b" (UID: "1b411531-02da-4593-a357-1a15f782c76b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:55.864140 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.864120    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-workload-certs" pod "1b411531-02da-4593-a357-1a15f782c76b" (UID: "1b411531-02da-4593-a357-1a15f782c76b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:55.864392 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.864370    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:55.864539 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.864526    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:55.864629 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.864618    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b411531-02da-4593-a357-1a15f782c76b-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:55.864711 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.864700    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:55.864797 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.864786    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:55.866330 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.866303    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b411531-02da-4593-a357-1a15f782c76b-kube-api-access-gxvfx" pod "1b411531-02da-4593-a357-1a15f782c76b" (UID: "1b411531-02da-4593-a357-1a15f782c76b"). InnerVolumeSpecName "kube-api-access-gxvfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:55.866718 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.866691    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-istio-envoy" pod "1b411531-02da-4593-a357-1a15f782c76b" (UID: "1b411531-02da-4593-a357-1a15f782c76b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:55.866718 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.866698    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b411531-02da-4593-a357-1a15f782c76b-istio-token" pod "1b411531-02da-4593-a357-1a15f782c76b" (UID: "1b411531-02da-4593-a357-1a15f782c76b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:55.866842 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.866769    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b411531-02da-4593-a357-1a15f782c76b-istio-podinfo" pod "1b411531-02da-4593-a357-1a15f782c76b" (UID: "1b411531-02da-4593-a357-1a15f782c76b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:05:55.882939 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.882915    5087 generic.go:356] "Generic (PLEG): container finished" podID="1b411531-02da-4593-a357-1a15f782c76b" containerID="21d90f075be8baa5083465e6c5d42b5d825eb96fc2a8d69ca0fe0f99bf5f6ab7" exitCode=0
Oct 30 22:05:55.883040 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.882971    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp" event={"ID":"1b411531-02da-4593-a357-1a15f782c76b","Type":"ContainerDied","Data":"21d90f075be8baa5083465e6c5d42b5d825eb96fc2a8d69ca0fe0f99bf5f6ab7"}
Oct 30 22:05:55.883040 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.882977    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"
Oct 30 22:05:55.883040 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.883023    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp" event={"ID":"1b411531-02da-4593-a357-1a15f782c76b","Type":"ContainerDied","Data":"15bdb23852a14e5b57d68d9e5f431d9bd9306c4d6541e1ce4798ba5f9618d55c"}
Oct 30 22:05:55.883156 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.883051    5087 scope.go:122] "RemoveContainer" containerID="21d90f075be8baa5083465e6c5d42b5d825eb96fc2a8d69ca0fe0f99bf5f6ab7"
Oct 30 22:05:55.897464 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.897448    5087 scope.go:122] "RemoveContainer" containerID="21d90f075be8baa5083465e6c5d42b5d825eb96fc2a8d69ca0fe0f99bf5f6ab7"
Oct 30 22:05:55.897720 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:05:55.897701    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21d90f075be8baa5083465e6c5d42b5d825eb96fc2a8d69ca0fe0f99bf5f6ab7\": container with ID starting with 21d90f075be8baa5083465e6c5d42b5d825eb96fc2a8d69ca0fe0f99bf5f6ab7 not found: ID does not exist" containerID="21d90f075be8baa5083465e6c5d42b5d825eb96fc2a8d69ca0fe0f99bf5f6ab7"
Oct 30 22:05:55.897776 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.897729    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d90f075be8baa5083465e6c5d42b5d825eb96fc2a8d69ca0fe0f99bf5f6ab7"} err="failed to get container status \"21d90f075be8baa5083465e6c5d42b5d825eb96fc2a8d69ca0fe0f99bf5f6ab7\": rpc error: code = NotFound desc = could not find container \"21d90f075be8baa5083465e6c5d42b5d825eb96fc2a8d69ca0fe0f99bf5f6ab7\": container with ID starting with 21d90f075be8baa5083465e6c5d42b5d825eb96fc2a8d69ca0fe0f99bf5f6ab7 not found: ID does not exist"
Oct 30 22:05:55.967416 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.967377    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b411531-02da-4593-a357-1a15f782c76b-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:55.967416 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.967413    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxvfx\" (UniqueName: \"kubernetes.io/projected/1b411531-02da-4593-a357-1a15f782c76b-kube-api-access-gxvfx\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:55.967645 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.967427    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b411531-02da-4593-a357-1a15f782c76b-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:55.967645 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:55.967436    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b411531-02da-4593-a357-1a15f782c76b-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:05:56.060399 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:56.060292    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"]
Oct 30 22:05:56.065591 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:56.065569    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-khbxp"]
Oct 30 22:05:57.709898 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:05:57.709865    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b411531-02da-4593-a357-1a15f782c76b" path="/var/lib/kubelet/pods/1b411531-02da-4593-a357-1a15f782c76b/volumes"
Oct 30 22:06:12.260501 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:12.260466    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk"]
Oct 30 22:06:12.309561 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:12.309528    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-e2deec91-nk9fk"]
Oct 30 22:06:13.710480 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:13.710443    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="294e4f05-6ceb-4441-b6dd-e9e197048ea6" path="/var/lib/kubelet/pods/294e4f05-6ceb-4441-b6dd-e9e197048ea6/volumes"
Oct 30 22:06:17.172588 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:06:17.172541    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:06:22.580487 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.580454    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"]
Oct 30 22:06:22.581112 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.581097    5087 cpu_manager.go:414] "containerMap: removing container" podUID="1b411531-02da-4593-a357-1a15f782c76b" containerName="istio-proxy"
Oct 30 22:06:22.581157 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.581114    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b411531-02da-4593-a357-1a15f782c76b" containerName="istio-proxy"
Oct 30 22:06:22.581157 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.581139    5087 cpu_manager.go:414] "containerMap: removing container" podUID="294e4f05-6ceb-4441-b6dd-e9e197048ea6" containerName="revoke-keys"
Oct 30 22:06:22.581157 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.581144    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="294e4f05-6ceb-4441-b6dd-e9e197048ea6" containerName="revoke-keys"
Oct 30 22:06:22.581257 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.581250    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b411531-02da-4593-a357-1a15f782c76b" containerName="istio-proxy"
Oct 30 22:06:22.581293 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.581261    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="294e4f05-6ceb-4441-b6dd-e9e197048ea6" containerName="revoke-keys"
Oct 30 22:06:22.584845 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.584829    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:22.612996 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.612962    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c975ae55-4f25-494b-acd0-7458374253df-istio-token\") pod \"e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") " pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:22.613527 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.613412    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c975ae55-4f25-494b-acd0-7458374253df-istiod-ca-cert\") pod \"e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") " pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:22.613739 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.613721    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-istio-envoy\") pod \"e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") " pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:22.613939 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.613924    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-credential-socket\") pod \"e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") " pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:22.614149 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.614088    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c975ae55-4f25-494b-acd0-7458374253df-istio-podinfo\") pod \"e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") " pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:22.614299 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.614283    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24swd\" (UniqueName: \"kubernetes.io/projected/c975ae55-4f25-494b-acd0-7458374253df-kube-api-access-24swd\") pod \"e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") " pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:22.617198 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.617151    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-workload-socket\") pod \"e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") " pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:22.617550 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.617506    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-workload-certs\") pod \"e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") " pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:22.618622 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.618599    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-istio-data\") pod \"e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") " pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:22.639376 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.639095    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9c8c4d-openshift-default-dockercfg-mnkzq\""
Oct 30 22:06:22.721416 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.713034    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"]
Oct 30 22:06:22.722100 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.722067    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-workload-socket\") pod \"e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") " pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:22.722472 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.722451    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-workload-certs\") pod \"e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") " pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:22.722705 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.722577    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-workload-socket\") pod \"e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") " pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:22.722823 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.722651    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-istio-data\") pod \"e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") " pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:22.722993 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.722830    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-workload-certs\") pod \"e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") " pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:22.723074 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.722989    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-istio-data\") pod \"e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") " pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:22.723611 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.723388    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c975ae55-4f25-494b-acd0-7458374253df-istio-token\") pod \"e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") " pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:22.723611 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.723447    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c975ae55-4f25-494b-acd0-7458374253df-istiod-ca-cert\") pod \"e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") " pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:22.723611 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.723476    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-istio-envoy\") pod \"e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") " pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:22.723611 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.723517    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-credential-socket\") pod \"e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") " pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:22.723611 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.723541    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c975ae55-4f25-494b-acd0-7458374253df-istio-podinfo\") pod \"e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") " pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:22.723611 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.723568    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24swd\" (UniqueName: \"kubernetes.io/projected/c975ae55-4f25-494b-acd0-7458374253df-kube-api-access-24swd\") pod \"e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") " pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:22.724397 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.724170    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-credential-socket\") pod \"e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") " pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:22.724855 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.724730    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c975ae55-4f25-494b-acd0-7458374253df-istiod-ca-cert\") pod \"e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") " pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:22.726453 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.726428    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-istio-envoy\") pod \"e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") " pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:22.749288 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.749223    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c975ae55-4f25-494b-acd0-7458374253df-istio-podinfo\") pod \"e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") " pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:22.780074 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.780044    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c975ae55-4f25-494b-acd0-7458374253df-istio-token\") pod \"e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") " pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:22.780241 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.780138    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24swd\" (UniqueName: \"kubernetes.io/projected/c975ae55-4f25-494b-acd0-7458374253df-kube-api-access-24swd\") pod \"e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") " pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:22.899697 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:22.899618    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:23.362797 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:23.362772    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"]
Oct 30 22:06:23.366257 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:06:23.366217    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc975ae55_4f25_494b_acd0_7458374253df.slice/crio-23457629204cbdfb787d8bdf87e6adfc1c5c2f2952bea7ea20308de0a4187a18 WatchSource:0}: Error finding container 23457629204cbdfb787d8bdf87e6adfc1c5c2f2952bea7ea20308de0a4187a18: Status 404 returned error can't find the container with id 23457629204cbdfb787d8bdf87e6adfc1c5c2f2952bea7ea20308de0a4187a18
Oct 30 22:06:23.369461 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:23.369421    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:06:23.369541 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:23.369512    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:06:23.369587 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:23.369557    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:06:24.110552 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:24.110517    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82" event={"ID":"c975ae55-4f25-494b-acd0-7458374253df","Type":"ContainerStarted","Data":"ed425bb89ff342c4a1061e01a0f7e2011ab1194812c0b379a84bdba67dd54487"}
Oct 30 22:06:24.111034 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:24.110574    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82" event={"ID":"c975ae55-4f25-494b-acd0-7458374253df","Type":"ContainerStarted","Data":"23457629204cbdfb787d8bdf87e6adfc1c5c2f2952bea7ea20308de0a4187a18"}
Oct 30 22:06:24.277416 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:24.277317    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82" podStartSLOduration=2.2772990650000002 podStartE2EDuration="2.277299065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:24.275318884 +0000 UTC m=+1978.749822862" watchObservedRunningTime="2026-10-30 22:06:24.277299065 +0000 UTC m=+1978.751803045"
Oct 30 22:06:24.900084 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:24.900051    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:24.906459 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:24.906437    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:25.117244 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:25.117219    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:25.118197 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:25.118181    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:06:28.756265 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:28.756234    5087 scope.go:122] "RemoveContainer" containerID="d454de01d64bbb57a0c0f6dafb88a1f65af1e9f2f8ff750b8e0df2c0c3a9aa7e"
Oct 30 22:06:28.766212 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:28.766195    5087 scope.go:122] "RemoveContainer" containerID="b86581bf3f90f338b9319b86f1a8244556673a4a21b513d3c0d73c4e0ec4e6c9"
Oct 30 22:06:28.775767 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:06:28.775747    5087 scope.go:122] "RemoveContainer" containerID="49ccadc6c6235eb7f0a2269db72f4d77c190c5d942a668f9a2c800a19da3d7f7"
Oct 30 22:06:47.172618 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:06:47.172576    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:07:02.531576 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:02.531538    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"]
Oct 30 22:07:02.532763 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:02.532724    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf" podUID="b2fda28b-f446-4917-bbe1-721ecff56fd8" containerName="istio-proxy" containerID="cri-o://2c1017b716a5455a38caaacb432a5cc3ac6b0f143a3e7fdc0fd853888137021b" gracePeriod=30
Oct 30 22:07:03.589737 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:03.589687    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-78cf83-796db64fff-7ph2p"]
Oct 30 22:07:03.609514 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:03.597567    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78cf83-796db64fff-7ph2p"
Oct 30 22:07:03.619285 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:03.619246    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab4753ec-050c-4e05-b558-cc77998d92a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-78cf83-796db64fff-7ph2p\" (UID: \"ab4753ec-050c-4e05-b558-cc77998d92a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-78cf83-796db64fff-7ph2p"
Oct 30 22:07:03.619520 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:03.619338    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-644z7\" (UniqueName: \"kubernetes.io/projected/ab4753ec-050c-4e05-b558-cc77998d92a6-kube-api-access-644z7\") pod \"payload-pre-processing-e2e-api-a-78cf83-796db64fff-7ph2p\" (UID: \"ab4753ec-050c-4e05-b558-cc77998d92a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-78cf83-796db64fff-7ph2p"
Oct 30 22:07:03.657178 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:03.657127    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-78cf83-796db64fff-7ph2p"]
Oct 30 22:07:03.657492 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:03.657471    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-78cf83\""
Oct 30 22:07:03.658143 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:03.658120    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-78cf83-dockercfg-dxqr6\""
Oct 30 22:07:03.762160 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:03.762120    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab4753ec-050c-4e05-b558-cc77998d92a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-78cf83-796db64fff-7ph2p\" (UID: \"ab4753ec-050c-4e05-b558-cc77998d92a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-78cf83-796db64fff-7ph2p"
Oct 30 22:07:03.762393 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:03.762255    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-644z7\" (UniqueName: \"kubernetes.io/projected/ab4753ec-050c-4e05-b558-cc77998d92a6-kube-api-access-644z7\") pod \"payload-pre-processing-e2e-api-a-78cf83-796db64fff-7ph2p\" (UID: \"ab4753ec-050c-4e05-b558-cc77998d92a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-78cf83-796db64fff-7ph2p"
Oct 30 22:07:03.765700 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:03.764554    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab4753ec-050c-4e05-b558-cc77998d92a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-78cf83-796db64fff-7ph2p\" (UID: \"ab4753ec-050c-4e05-b558-cc77998d92a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-78cf83-796db64fff-7ph2p"
Oct 30 22:07:03.823749 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:03.823717    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-644z7\" (UniqueName: \"kubernetes.io/projected/ab4753ec-050c-4e05-b558-cc77998d92a6-kube-api-access-644z7\") pod \"payload-pre-processing-e2e-api-a-78cf83-796db64fff-7ph2p\" (UID: \"ab4753ec-050c-4e05-b558-cc77998d92a6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-78cf83-796db64fff-7ph2p"
Oct 30 22:07:03.934640 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:03.934548    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78cf83-796db64fff-7ph2p"
Oct 30 22:07:04.322412 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:04.322382    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-78cf83-796db64fff-7ph2p"]
Oct 30 22:07:04.323004 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:07:04.322982    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab4753ec_050c_4e05_b558_cc77998d92a6.slice/crio-d9c14a89be6f26e6981e9d47d8ceb3749793fbb6474af601cc6e42361340bcfc WatchSource:0}: Error finding container d9c14a89be6f26e6981e9d47d8ceb3749793fbb6474af601cc6e42361340bcfc: Status 404 returned error can't find the container with id d9c14a89be6f26e6981e9d47d8ceb3749793fbb6474af601cc6e42361340bcfc
Oct 30 22:07:04.415956 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:04.415917    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78cf83-796db64fff-7ph2p" event={"ID":"ab4753ec-050c-4e05-b558-cc77998d92a6","Type":"ContainerStarted","Data":"c3e6f05d371590674117d3cee45591d9d3e1d997f263da7819d32b3e0a9f1197"}
Oct 30 22:07:04.416099 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:04.415984    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78cf83-796db64fff-7ph2p" event={"ID":"ab4753ec-050c-4e05-b558-cc77998d92a6","Type":"ContainerStarted","Data":"d9c14a89be6f26e6981e9d47d8ceb3749793fbb6474af601cc6e42361340bcfc"}
Oct 30 22:07:04.416099 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:04.416011    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78cf83-796db64fff-7ph2p"
Oct 30 22:07:04.533289 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:04.533240    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78cf83-796db64fff-7ph2p" podStartSLOduration=1.533224243 podStartE2EDuration="1.533224243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:04.531086758 +0000 UTC m=+2019.005590758" watchObservedRunningTime="2026-10-30 22:07:04.533224243 +0000 UTC m=+2019.007728228"
Oct 30 22:07:05.756115 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:05.756078    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"]
Oct 30 22:07:05.756636 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:05.756288    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf" podUID="66ef40b8-03ce-449b-ab82-da0b7af9e155" containerName="istio-proxy" containerID="cri-o://0015000afa9c7705810d6c864ba040b04ec2ff91cbfed8ba9920266178cc9f7b" gracePeriod=30
Oct 30 22:07:06.082232 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:06.082131    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"]
Oct 30 22:07:06.082563 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:06.082525    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc" podUID="322d51a7-324e-4f33-a606-3384bb72d3cc" containerName="istio-proxy" containerID="cri-o://083569d08e187f3975f006cc2162191cc791ca68796421440dd8ff3e28df75cf" gracePeriod=30
Oct 30 22:07:06.086889 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:06.086860    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"]
Oct 30 22:07:06.087402 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:06.087128    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk" podUID="6f891215-2cc6-4b56-8ba2-68687e58f21c" containerName="istio-proxy" containerID="cri-o://d5bc6fca893b137116b81e4aa445275f19416b0b9faad4743e2ca6d3b960bebb" gracePeriod=30
Oct 30 22:07:06.090866 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:06.090813    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"]
Oct 30 22:07:06.091104 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:06.091065    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv" podUID="bc565097-2279-4f89-a2a6-35e117fd929a" containerName="istio-proxy" containerID="cri-o://3cc130c3ff474265a6b8dd8c8353b5ba573d5d7ed8469e2b235d7b9ba8beefa9" gracePeriod=30
Oct 30 22:07:06.095156 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:06.095129    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"]
Oct 30 22:07:06.095420 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:06.095382    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc" podUID="62fa8024-dac6-4993-b671-23bda513f309" containerName="istio-proxy" containerID="cri-o://be67215a5cbffa912d1a3de9277cf14d96e9435b5c1ebe523d7461be348c7bcb" gracePeriod=30
Oct 30 22:07:06.131767 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:06.130965    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"]
Oct 30 22:07:06.131767 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:06.131219    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl" podUID="061e2d92-1dd6-4373-b93a-09dd58d23a0e" containerName="istio-proxy" containerID="cri-o://94e5dfb2d66b0be6a057524b68d8131c4521f2cc668683d3cdcc993e11ff2650" gracePeriod=30
Oct 30 22:07:06.136592 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:06.136557    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"]
Oct 30 22:07:06.136832 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:06.136806    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd" podUID="280b7c38-6ad1-4e93-ba46-ba4ae8171ca9" containerName="istio-proxy" containerID="cri-o://fb3deb51b4b87b4e24ff09d3aab21c372977abc28aec7479efca379e39630308" gracePeriod=30
Oct 30 22:07:06.259852 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:06.259818    5087 patch_prober.go:29] interesting pod/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:07:06.260027 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:06.259879    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf" podUID="b2fda28b-f446-4917-bbe1-721ecff56fd8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:07:06.798024 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:06.797980    5087 patch_prober.go:29] interesting pod/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:07:06.798443 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:06.798038    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf" podUID="66ef40b8-03ce-449b-ab82-da0b7af9e155" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:07:07.853480 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:07.853456    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 22:07:07.950910 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:07.950883    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2fda28b-f446-4917-bbe1-721ecff56fd8-istio-token\" (UniqueName: \"kubernetes.io/projected/b2fda28b-f446-4917-bbe1-721ecff56fd8-istio-token\") pod \"b2fda28b-f446-4917-bbe1-721ecff56fd8\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") "
Oct 30 22:07:07.951105 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:07.950919    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-workload-socket\") pod \"b2fda28b-f446-4917-bbe1-721ecff56fd8\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") "
Oct 30 22:07:07.951105 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:07.950987    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2fda28b-f446-4917-bbe1-721ecff56fd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2fda28b-f446-4917-bbe1-721ecff56fd8-istiod-ca-cert\") pod \"b2fda28b-f446-4917-bbe1-721ecff56fd8\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") "
Oct 30 22:07:07.951105 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:07.951063    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-workload-certs\") pod \"b2fda28b-f446-4917-bbe1-721ecff56fd8\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") "
Oct 30 22:07:07.951105 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:07.951093    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-credential-socket\") pod \"b2fda28b-f446-4917-bbe1-721ecff56fd8\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") "
Oct 30 22:07:07.951335 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:07.951148    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-istio-envoy\") pod \"b2fda28b-f446-4917-bbe1-721ecff56fd8\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") "
Oct 30 22:07:07.951335 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:07.951227    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2fda28b-f446-4917-bbe1-721ecff56fd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2fda28b-f446-4917-bbe1-721ecff56fd8-istio-podinfo\") pod \"b2fda28b-f446-4917-bbe1-721ecff56fd8\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") "
Oct 30 22:07:07.951335 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:07.951259    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-workload-socket" pod "b2fda28b-f446-4917-bbe1-721ecff56fd8" (UID: "b2fda28b-f446-4917-bbe1-721ecff56fd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:07.951555 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:07.951417    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-istio-data\") pod \"b2fda28b-f446-4917-bbe1-721ecff56fd8\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") "
Oct 30 22:07:07.951555 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:07.951430    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-workload-certs" pod "b2fda28b-f446-4917-bbe1-721ecff56fd8" (UID: "b2fda28b-f446-4917-bbe1-721ecff56fd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:07.951555 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:07.951447    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-credential-socket" pod "b2fda28b-f446-4917-bbe1-721ecff56fd8" (UID: "b2fda28b-f446-4917-bbe1-721ecff56fd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:07.951555 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:07.951454    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2fda28b-f446-4917-bbe1-721ecff56fd8-kube-api-access-dhp4x\" (UniqueName: \"kubernetes.io/projected/b2fda28b-f446-4917-bbe1-721ecff56fd8-kube-api-access-dhp4x\") pod \"b2fda28b-f446-4917-bbe1-721ecff56fd8\" (UID: \"b2fda28b-f446-4917-bbe1-721ecff56fd8\") "
Oct 30 22:07:07.951555 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:07.951485    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2fda28b-f446-4917-bbe1-721ecff56fd8-istiod-ca-cert" pod "b2fda28b-f446-4917-bbe1-721ecff56fd8" (UID: "b2fda28b-f446-4917-bbe1-721ecff56fd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:07.951984 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:07.951665    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-istio-data" pod "b2fda28b-f446-4917-bbe1-721ecff56fd8" (UID: "b2fda28b-f446-4917-bbe1-721ecff56fd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:07.952461 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:07.952435    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:07.952561 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:07.952467    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:07.952561 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:07.952484    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:07.952561 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:07.952498    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:07.952561 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:07.952512    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2fda28b-f446-4917-bbe1-721ecff56fd8-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:07.953557 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:07.953531    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-istio-envoy" pod "b2fda28b-f446-4917-bbe1-721ecff56fd8" (UID: "b2fda28b-f446-4917-bbe1-721ecff56fd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:07.953656 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:07.953550    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2fda28b-f446-4917-bbe1-721ecff56fd8-istio-podinfo" pod "b2fda28b-f446-4917-bbe1-721ecff56fd8" (UID: "b2fda28b-f446-4917-bbe1-721ecff56fd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:07:07.953656 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:07.953576    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2fda28b-f446-4917-bbe1-721ecff56fd8-istio-token" pod "b2fda28b-f446-4917-bbe1-721ecff56fd8" (UID: "b2fda28b-f446-4917-bbe1-721ecff56fd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:07.953656 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:07.953629    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2fda28b-f446-4917-bbe1-721ecff56fd8-kube-api-access-dhp4x" pod "b2fda28b-f446-4917-bbe1-721ecff56fd8" (UID: "b2fda28b-f446-4917-bbe1-721ecff56fd8"). InnerVolumeSpecName "kube-api-access-dhp4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:08.054325 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:08.054288    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2fda28b-f446-4917-bbe1-721ecff56fd8-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:08.054325 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:08.054327    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2fda28b-f446-4917-bbe1-721ecff56fd8-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:08.054556 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:08.054360    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhp4x\" (UniqueName: \"kubernetes.io/projected/b2fda28b-f446-4917-bbe1-721ecff56fd8-kube-api-access-dhp4x\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:08.054556 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:08.054378    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2fda28b-f446-4917-bbe1-721ecff56fd8-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:08.447092 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:08.447061    5087 generic.go:356] "Generic (PLEG): container finished" podID="b2fda28b-f446-4917-bbe1-721ecff56fd8" containerID="2c1017b716a5455a38caaacb432a5cc3ac6b0f143a3e7fdc0fd853888137021b" exitCode=0
Oct 30 22:07:08.447289 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:08.447132    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"
Oct 30 22:07:08.447289 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:08.447150    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf" event={"ID":"b2fda28b-f446-4917-bbe1-721ecff56fd8","Type":"ContainerDied","Data":"2c1017b716a5455a38caaacb432a5cc3ac6b0f143a3e7fdc0fd853888137021b"}
Oct 30 22:07:08.447289 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:08.447205    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf" event={"ID":"b2fda28b-f446-4917-bbe1-721ecff56fd8","Type":"ContainerDied","Data":"7a688b283a4064c482532cc09e22a8d5593cbad2e02ba1652a5e4722dcb4739f"}
Oct 30 22:07:08.447289 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:08.447228    5087 scope.go:122] "RemoveContainer" containerID="2c1017b716a5455a38caaacb432a5cc3ac6b0f143a3e7fdc0fd853888137021b"
Oct 30 22:07:08.459228 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:08.459206    5087 scope.go:122] "RemoveContainer" containerID="2c1017b716a5455a38caaacb432a5cc3ac6b0f143a3e7fdc0fd853888137021b"
Oct 30 22:07:08.459564 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:07:08.459540    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c1017b716a5455a38caaacb432a5cc3ac6b0f143a3e7fdc0fd853888137021b\": container with ID starting with 2c1017b716a5455a38caaacb432a5cc3ac6b0f143a3e7fdc0fd853888137021b not found: ID does not exist" containerID="2c1017b716a5455a38caaacb432a5cc3ac6b0f143a3e7fdc0fd853888137021b"
Oct 30 22:07:08.459664 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:08.459572    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c1017b716a5455a38caaacb432a5cc3ac6b0f143a3e7fdc0fd853888137021b"} err="failed to get container status \"2c1017b716a5455a38caaacb432a5cc3ac6b0f143a3e7fdc0fd853888137021b\": rpc error: code = NotFound desc = could not find container \"2c1017b716a5455a38caaacb432a5cc3ac6b0f143a3e7fdc0fd853888137021b\": container with ID starting with 2c1017b716a5455a38caaacb432a5cc3ac6b0f143a3e7fdc0fd853888137021b not found: ID does not exist"
Oct 30 22:07:08.565922 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:08.565884    5087 patch_prober.go:29] interesting pod/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:07:08.566097 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:08.565956    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc" podUID="322d51a7-324e-4f33-a606-3384bb72d3cc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:07:08.612137 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:08.612104    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"]
Oct 30 22:07:08.663141 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:08.663103    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-tmlmf"]
Oct 30 22:07:09.711129 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:09.711100    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2fda28b-f446-4917-bbe1-721ecff56fd8" path="/var/lib/kubelet/pods/b2fda28b-f446-4917-bbe1-721ecff56fd8/volumes"
Oct 30 22:07:10.085074 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:10.085040    5087 patch_prober.go:29] interesting pod/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:07:10.085245 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:10.085095    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk" podUID="6f891215-2cc6-4b56-8ba2-68687e58f21c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:07:11.052689 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.052653    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 22:07:11.210866 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.210837    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66ef40b8-03ce-449b-ab82-da0b7af9e155-kube-api-access-5rhz7\" (UniqueName: \"kubernetes.io/projected/66ef40b8-03ce-449b-ab82-da0b7af9e155-kube-api-access-5rhz7\") pod \"66ef40b8-03ce-449b-ab82-da0b7af9e155\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") "
Oct 30 22:07:11.210984 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.210927    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66ef40b8-03ce-449b-ab82-da0b7af9e155-istio-token\" (UniqueName: \"kubernetes.io/projected/66ef40b8-03ce-449b-ab82-da0b7af9e155-istio-token\") pod \"66ef40b8-03ce-449b-ab82-da0b7af9e155\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") "
Oct 30 22:07:11.210984 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.210965    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-workload-socket\") pod \"66ef40b8-03ce-449b-ab82-da0b7af9e155\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") "
Oct 30 22:07:11.211064 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.211044    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-istio-data\") pod \"66ef40b8-03ce-449b-ab82-da0b7af9e155\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") "
Oct 30 22:07:11.211124 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.211107    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-workload-certs\") pod \"66ef40b8-03ce-449b-ab82-da0b7af9e155\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") "
Oct 30 22:07:11.211269 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.211141    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-istio-envoy\") pod \"66ef40b8-03ce-449b-ab82-da0b7af9e155\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") "
Oct 30 22:07:11.211269 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.211191    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-credential-socket\") pod \"66ef40b8-03ce-449b-ab82-da0b7af9e155\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") "
Oct 30 22:07:11.211269 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.211230    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66ef40b8-03ce-449b-ab82-da0b7af9e155-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ef40b8-03ce-449b-ab82-da0b7af9e155-istiod-ca-cert\") pod \"66ef40b8-03ce-449b-ab82-da0b7af9e155\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") "
Oct 30 22:07:11.211269 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.211233    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-workload-socket" pod "66ef40b8-03ce-449b-ab82-da0b7af9e155" (UID: "66ef40b8-03ce-449b-ab82-da0b7af9e155"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.211725 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.211335    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-istio-data" pod "66ef40b8-03ce-449b-ab82-da0b7af9e155" (UID: "66ef40b8-03ce-449b-ab82-da0b7af9e155"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.211725 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.211377    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66ef40b8-03ce-449b-ab82-da0b7af9e155-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ef40b8-03ce-449b-ab82-da0b7af9e155-istio-podinfo\") pod \"66ef40b8-03ce-449b-ab82-da0b7af9e155\" (UID: \"66ef40b8-03ce-449b-ab82-da0b7af9e155\") "
Oct 30 22:07:11.211725 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.211529    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-credential-socket" pod "66ef40b8-03ce-449b-ab82-da0b7af9e155" (UID: "66ef40b8-03ce-449b-ab82-da0b7af9e155"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.211725 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.211544    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-workload-certs" pod "66ef40b8-03ce-449b-ab82-da0b7af9e155" (UID: "66ef40b8-03ce-449b-ab82-da0b7af9e155"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.211725 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.211619    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66ef40b8-03ce-449b-ab82-da0b7af9e155-istiod-ca-cert" pod "66ef40b8-03ce-449b-ab82-da0b7af9e155" (UID: "66ef40b8-03ce-449b-ab82-da0b7af9e155"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:11.212296 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.212270    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.212405 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.212301    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.212405 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.212316    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66ef40b8-03ce-449b-ab82-da0b7af9e155-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.212405 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.212330    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.212405 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.212364    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.213195 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.213174    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ef40b8-03ce-449b-ab82-da0b7af9e155-kube-api-access-5rhz7" pod "66ef40b8-03ce-449b-ab82-da0b7af9e155" (UID: "66ef40b8-03ce-449b-ab82-da0b7af9e155"). InnerVolumeSpecName "kube-api-access-5rhz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:11.213285 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.213203    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66ef40b8-03ce-449b-ab82-da0b7af9e155-istio-token" pod "66ef40b8-03ce-449b-ab82-da0b7af9e155" (UID: "66ef40b8-03ce-449b-ab82-da0b7af9e155"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:11.213399 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.213381    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66ef40b8-03ce-449b-ab82-da0b7af9e155-istio-podinfo" pod "66ef40b8-03ce-449b-ab82-da0b7af9e155" (UID: "66ef40b8-03ce-449b-ab82-da0b7af9e155"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:07:11.213622 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.213606    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-istio-envoy" pod "66ef40b8-03ce-449b-ab82-da0b7af9e155" (UID: "66ef40b8-03ce-449b-ab82-da0b7af9e155"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.314231 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.314033    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66ef40b8-03ce-449b-ab82-da0b7af9e155-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.314231 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.314076    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rhz7\" (UniqueName: \"kubernetes.io/projected/66ef40b8-03ce-449b-ab82-da0b7af9e155-kube-api-access-5rhz7\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.314231 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.314093    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66ef40b8-03ce-449b-ab82-da0b7af9e155-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.314231 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.314107    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66ef40b8-03ce-449b-ab82-da0b7af9e155-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.472181 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.472141    5087 generic.go:356] "Generic (PLEG): container finished" podID="322d51a7-324e-4f33-a606-3384bb72d3cc" containerID="083569d08e187f3975f006cc2162191cc791ca68796421440dd8ff3e28df75cf" exitCode=0
Oct 30 22:07:11.472336 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.472267    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc" event={"ID":"322d51a7-324e-4f33-a606-3384bb72d3cc","Type":"ContainerDied","Data":"083569d08e187f3975f006cc2162191cc791ca68796421440dd8ff3e28df75cf"}
Oct 30 22:07:11.475811 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.474284    5087 generic.go:356] "Generic (PLEG): container finished" podID="061e2d92-1dd6-4373-b93a-09dd58d23a0e" containerID="94e5dfb2d66b0be6a057524b68d8131c4521f2cc668683d3cdcc993e11ff2650" exitCode=0
Oct 30 22:07:11.475811 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.474361    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl" event={"ID":"061e2d92-1dd6-4373-b93a-09dd58d23a0e","Type":"ContainerDied","Data":"94e5dfb2d66b0be6a057524b68d8131c4521f2cc668683d3cdcc993e11ff2650"}
Oct 30 22:07:11.481282 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.481259    5087 generic.go:356] "Generic (PLEG): container finished" podID="6f891215-2cc6-4b56-8ba2-68687e58f21c" containerID="d5bc6fca893b137116b81e4aa445275f19416b0b9faad4743e2ca6d3b960bebb" exitCode=0
Oct 30 22:07:11.481519 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.481323    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk" event={"ID":"6f891215-2cc6-4b56-8ba2-68687e58f21c","Type":"ContainerDied","Data":"d5bc6fca893b137116b81e4aa445275f19416b0b9faad4743e2ca6d3b960bebb"}
Oct 30 22:07:11.483691 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.483664    5087 generic.go:356] "Generic (PLEG): container finished" podID="62fa8024-dac6-4993-b671-23bda513f309" containerID="be67215a5cbffa912d1a3de9277cf14d96e9435b5c1ebe523d7461be348c7bcb" exitCode=0
Oct 30 22:07:11.483808 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.483748    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc" event={"ID":"62fa8024-dac6-4993-b671-23bda513f309","Type":"ContainerDied","Data":"be67215a5cbffa912d1a3de9277cf14d96e9435b5c1ebe523d7461be348c7bcb"}
Oct 30 22:07:11.486442 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.486417    5087 generic.go:356] "Generic (PLEG): container finished" podID="bc565097-2279-4f89-a2a6-35e117fd929a" containerID="3cc130c3ff474265a6b8dd8c8353b5ba573d5d7ed8469e2b235d7b9ba8beefa9" exitCode=0
Oct 30 22:07:11.486667 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.486643    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv" event={"ID":"bc565097-2279-4f89-a2a6-35e117fd929a","Type":"ContainerDied","Data":"3cc130c3ff474265a6b8dd8c8353b5ba573d5d7ed8469e2b235d7b9ba8beefa9"}
Oct 30 22:07:11.488249 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.488219    5087 generic.go:356] "Generic (PLEG): container finished" podID="66ef40b8-03ce-449b-ab82-da0b7af9e155" containerID="0015000afa9c7705810d6c864ba040b04ec2ff91cbfed8ba9920266178cc9f7b" exitCode=0
Oct 30 22:07:11.488375 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.488299    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"
Oct 30 22:07:11.488441 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.488408    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf" event={"ID":"66ef40b8-03ce-449b-ab82-da0b7af9e155","Type":"ContainerDied","Data":"0015000afa9c7705810d6c864ba040b04ec2ff91cbfed8ba9920266178cc9f7b"}
Oct 30 22:07:11.488510 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.488451    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf" event={"ID":"66ef40b8-03ce-449b-ab82-da0b7af9e155","Type":"ContainerDied","Data":"7e26218d04f2a3dfc8cd3269e58b2dde181e1447950d427a5863c10ba8df8690"}
Oct 30 22:07:11.488510 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.488482    5087 scope.go:122] "RemoveContainer" containerID="0015000afa9c7705810d6c864ba040b04ec2ff91cbfed8ba9920266178cc9f7b"
Oct 30 22:07:11.491720 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.491697    5087 generic.go:356] "Generic (PLEG): container finished" podID="280b7c38-6ad1-4e93-ba46-ba4ae8171ca9" containerID="fb3deb51b4b87b4e24ff09d3aab21c372977abc28aec7479efca379e39630308" exitCode=0
Oct 30 22:07:11.491803 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.491740    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd" event={"ID":"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9","Type":"ContainerDied","Data":"fb3deb51b4b87b4e24ff09d3aab21c372977abc28aec7479efca379e39630308"}
Oct 30 22:07:11.510978 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.507097    5087 scope.go:122] "RemoveContainer" containerID="0015000afa9c7705810d6c864ba040b04ec2ff91cbfed8ba9920266178cc9f7b"
Oct 30 22:07:11.510978 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:07:11.508715    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0015000afa9c7705810d6c864ba040b04ec2ff91cbfed8ba9920266178cc9f7b\": container with ID starting with 0015000afa9c7705810d6c864ba040b04ec2ff91cbfed8ba9920266178cc9f7b not found: ID does not exist" containerID="0015000afa9c7705810d6c864ba040b04ec2ff91cbfed8ba9920266178cc9f7b"
Oct 30 22:07:11.510978 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.508758    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0015000afa9c7705810d6c864ba040b04ec2ff91cbfed8ba9920266178cc9f7b"} err="failed to get container status \"0015000afa9c7705810d6c864ba040b04ec2ff91cbfed8ba9920266178cc9f7b\": rpc error: code = NotFound desc = could not find container \"0015000afa9c7705810d6c864ba040b04ec2ff91cbfed8ba9920266178cc9f7b\": container with ID starting with 0015000afa9c7705810d6c864ba040b04ec2ff91cbfed8ba9920266178cc9f7b not found: ID does not exist"
Oct 30 22:07:11.529441 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.526785    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 22:07:11.605370 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.605333    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 22:07:11.609495 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.609469    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 22:07:11.615077 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.615055    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 22:07:11.625622 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.625599    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-istio-envoy\") pod \"322d51a7-324e-4f33-a606-3384bb72d3cc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") "
Oct 30 22:07:11.625735 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.625675    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-workload-certs\") pod \"322d51a7-324e-4f33-a606-3384bb72d3cc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") "
Oct 30 22:07:11.625806 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.625739    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-istio-data\") pod \"322d51a7-324e-4f33-a606-3384bb72d3cc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") "
Oct 30 22:07:11.625864 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.625811    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/322d51a7-324e-4f33-a606-3384bb72d3cc-istio-token\" (UniqueName: \"kubernetes.io/projected/322d51a7-324e-4f33-a606-3384bb72d3cc-istio-token\") pod \"322d51a7-324e-4f33-a606-3384bb72d3cc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") "
Oct 30 22:07:11.625864 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.625834    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-workload-socket\") pod \"322d51a7-324e-4f33-a606-3384bb72d3cc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") "
Oct 30 22:07:11.625969 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.625872    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/322d51a7-324e-4f33-a606-3384bb72d3cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/322d51a7-324e-4f33-a606-3384bb72d3cc-istiod-ca-cert\") pod \"322d51a7-324e-4f33-a606-3384bb72d3cc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") "
Oct 30 22:07:11.625969 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.625899    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-credential-socket\") pod \"322d51a7-324e-4f33-a606-3384bb72d3cc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") "
Oct 30 22:07:11.625969 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.625943    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/322d51a7-324e-4f33-a606-3384bb72d3cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/322d51a7-324e-4f33-a606-3384bb72d3cc-istio-podinfo\") pod \"322d51a7-324e-4f33-a606-3384bb72d3cc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") "
Oct 30 22:07:11.626121 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.625984    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/322d51a7-324e-4f33-a606-3384bb72d3cc-kube-api-access-xwwcr\" (UniqueName: \"kubernetes.io/projected/322d51a7-324e-4f33-a606-3384bb72d3cc-kube-api-access-xwwcr\") pod \"322d51a7-324e-4f33-a606-3384bb72d3cc\" (UID: \"322d51a7-324e-4f33-a606-3384bb72d3cc\") "
Oct 30 22:07:11.626863 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.626835    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-workload-certs" pod "322d51a7-324e-4f33-a606-3384bb72d3cc" (UID: "322d51a7-324e-4f33-a606-3384bb72d3cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.627090 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.627053    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-credential-socket" pod "322d51a7-324e-4f33-a606-3384bb72d3cc" (UID: "322d51a7-324e-4f33-a606-3384bb72d3cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.627233 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.627207    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-workload-socket" pod "322d51a7-324e-4f33-a606-3384bb72d3cc" (UID: "322d51a7-324e-4f33-a606-3384bb72d3cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.627590 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.627557    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-istio-data" pod "322d51a7-324e-4f33-a606-3384bb72d3cc" (UID: "322d51a7-324e-4f33-a606-3384bb72d3cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.627590 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.627586    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/322d51a7-324e-4f33-a606-3384bb72d3cc-istiod-ca-cert" pod "322d51a7-324e-4f33-a606-3384bb72d3cc" (UID: "322d51a7-324e-4f33-a606-3384bb72d3cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:11.631752 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.631731    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322d51a7-324e-4f33-a606-3384bb72d3cc-kube-api-access-xwwcr" pod "322d51a7-324e-4f33-a606-3384bb72d3cc" (UID: "322d51a7-324e-4f33-a606-3384bb72d3cc"). InnerVolumeSpecName "kube-api-access-xwwcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:11.632539 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.632413    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-istio-envoy" pod "322d51a7-324e-4f33-a606-3384bb72d3cc" (UID: "322d51a7-324e-4f33-a606-3384bb72d3cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.633519 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.633490    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/322d51a7-324e-4f33-a606-3384bb72d3cc-istio-podinfo" pod "322d51a7-324e-4f33-a606-3384bb72d3cc" (UID: "322d51a7-324e-4f33-a606-3384bb72d3cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:07:11.634699 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.634667    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322d51a7-324e-4f33-a606-3384bb72d3cc-istio-token" pod "322d51a7-324e-4f33-a606-3384bb72d3cc" (UID: "322d51a7-324e-4f33-a606-3384bb72d3cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:11.656488 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.656460    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"]
Oct 30 22:07:11.663942 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.663911    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-fkzmf"]
Oct 30 22:07:11.687620 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.687595    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 22:07:11.711121 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.711095    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 22:07:11.711509 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.711489    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66ef40b8-03ce-449b-ab82-da0b7af9e155" path="/var/lib/kubelet/pods/66ef40b8-03ce-449b-ab82-da0b7af9e155/volumes"
Oct 30 22:07:11.726780 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.726754    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istio-podinfo\") pod \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") "
Oct 30 22:07:11.726887 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.726787    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-kube-api-access-d8bss\" (UniqueName: \"kubernetes.io/projected/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-kube-api-access-d8bss\") pod \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") "
Oct 30 22:07:11.726887 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.726813    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-workload-socket\") pod \"6f891215-2cc6-4b56-8ba2-68687e58f21c\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") "
Oct 30 22:07:11.726887 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.726855    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f891215-2cc6-4b56-8ba2-68687e58f21c-kube-api-access-8zlwm\" (UniqueName: \"kubernetes.io/projected/6f891215-2cc6-4b56-8ba2-68687e58f21c-kube-api-access-8zlwm\") pod \"6f891215-2cc6-4b56-8ba2-68687e58f21c\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") "
Oct 30 22:07:11.727070 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.726917    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-istio-data\") pod \"6f891215-2cc6-4b56-8ba2-68687e58f21c\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") "
Oct 30 22:07:11.727070 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.726982    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-workload-certs\") pod \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") "
Oct 30 22:07:11.727070 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.727044    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istio-token\" (UniqueName: \"kubernetes.io/projected/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istio-token\") pod \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") "
Oct 30 22:07:11.727222 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.727097    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-workload-socket" pod "6f891215-2cc6-4b56-8ba2-68687e58f21c" (UID: "6f891215-2cc6-4b56-8ba2-68687e58f21c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.727222 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.727105    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istio-data\") pod \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") "
Oct 30 22:07:11.727222 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.727178    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-workload-certs\") pod \"6f891215-2cc6-4b56-8ba2-68687e58f21c\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") "
Oct 30 22:07:11.727331 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.727222    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istio-podinfo\") pod \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") "
Oct 30 22:07:11.727331 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.727290    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f891215-2cc6-4b56-8ba2-68687e58f21c-istio-token\" (UniqueName: \"kubernetes.io/projected/6f891215-2cc6-4b56-8ba2-68687e58f21c-istio-token\") pod \"6f891215-2cc6-4b56-8ba2-68687e58f21c\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") "
Oct 30 22:07:11.727331 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.727313    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istio-data" pod "061e2d92-1dd6-4373-b93a-09dd58d23a0e" (UID: "061e2d92-1dd6-4373-b93a-09dd58d23a0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.727483 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.727325    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-istio-data" pod "6f891215-2cc6-4b56-8ba2-68687e58f21c" (UID: "6f891215-2cc6-4b56-8ba2-68687e58f21c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.727483 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.727317    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istio-data\") pod \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") "
Oct 30 22:07:11.727483 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.727430    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-workload-socket\") pod \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") "
Oct 30 22:07:11.727597 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.727514    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istiod-ca-cert\") pod \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") "
Oct 30 22:07:11.727597 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.727533    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-workload-certs" pod "061e2d92-1dd6-4373-b93a-09dd58d23a0e" (UID: "061e2d92-1dd6-4373-b93a-09dd58d23a0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.727597 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.727551    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-workload-certs\") pod \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") "
Oct 30 22:07:11.727597 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.727562    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istio-data" pod "280b7c38-6ad1-4e93-ba46-ba4ae8171ca9" (UID: "280b7c38-6ad1-4e93-ba46-ba4ae8171ca9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.727799 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.727598    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-credential-socket\") pod \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") "
Oct 30 22:07:11.727799 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.727639    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f891215-2cc6-4b56-8ba2-68687e58f21c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f891215-2cc6-4b56-8ba2-68687e58f21c-istio-podinfo\") pod \"6f891215-2cc6-4b56-8ba2-68687e58f21c\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") "
Oct 30 22:07:11.727799 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.727681    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-credential-socket\") pod \"6f891215-2cc6-4b56-8ba2-68687e58f21c\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") "
Oct 30 22:07:11.727799 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.727711    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-workload-socket\") pod \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") "
Oct 30 22:07:11.727799 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.727756    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istio-token\" (UniqueName: \"kubernetes.io/projected/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istio-token\") pod \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") "
Oct 30 22:07:11.727799 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.727773    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-workload-certs" pod "6f891215-2cc6-4b56-8ba2-68687e58f21c" (UID: "6f891215-2cc6-4b56-8ba2-68687e58f21c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.727799 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.727782    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istiod-ca-cert\") pod \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") "
Oct 30 22:07:11.728170 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.727839    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-istio-envoy\") pod \"6f891215-2cc6-4b56-8ba2-68687e58f21c\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") "
Oct 30 22:07:11.728170 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.727888    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istio-envoy\") pod \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\" (UID: \"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9\") "
Oct 30 22:07:11.728170 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.727941    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/061e2d92-1dd6-4373-b93a-09dd58d23a0e-kube-api-access-j8m2v\" (UniqueName: \"kubernetes.io/projected/061e2d92-1dd6-4373-b93a-09dd58d23a0e-kube-api-access-j8m2v\") pod \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") "
Oct 30 22:07:11.728170 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.728017    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f891215-2cc6-4b56-8ba2-68687e58f21c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f891215-2cc6-4b56-8ba2-68687e58f21c-istiod-ca-cert\") pod \"6f891215-2cc6-4b56-8ba2-68687e58f21c\" (UID: \"6f891215-2cc6-4b56-8ba2-68687e58f21c\") "
Oct 30 22:07:11.728170 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.728064    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istio-envoy\") pod \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") "
Oct 30 22:07:11.728170 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.728093    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-credential-socket\") pod \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\" (UID: \"061e2d92-1dd6-4373-b93a-09dd58d23a0e\") "
Oct 30 22:07:11.728826 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.728732    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/322d51a7-324e-4f33-a606-3384bb72d3cc-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.728826 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.728757    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.728826 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.728771    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwwcr\" (UniqueName: \"kubernetes.io/projected/322d51a7-324e-4f33-a606-3384bb72d3cc-kube-api-access-xwwcr\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.728826 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.728786    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.728826 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.728800    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.728826 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.728813    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.728826 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.728827    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.728826 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.728841    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.728826 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.728855    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.728826 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.728869    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/322d51a7-324e-4f33-a606-3384bb72d3cc-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.728826 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.728882    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.728826 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.728895    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.730324 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.728910    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/322d51a7-324e-4f33-a606-3384bb72d3cc-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.730324 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.728923    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.730324 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.728936    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/322d51a7-324e-4f33-a606-3384bb72d3cc-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.730825 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.728038    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istiod-ca-cert" pod "280b7c38-6ad1-4e93-ba46-ba4ae8171ca9" (UID: "280b7c38-6ad1-4e93-ba46-ba4ae8171ca9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:11.731114 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.729327    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-workload-socket" pod "061e2d92-1dd6-4373-b93a-09dd58d23a0e" (UID: "061e2d92-1dd6-4373-b93a-09dd58d23a0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.731207 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.729340    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-credential-socket" pod "061e2d92-1dd6-4373-b93a-09dd58d23a0e" (UID: "061e2d92-1dd6-4373-b93a-09dd58d23a0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.731328 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.729564    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-workload-certs" pod "280b7c38-6ad1-4e93-ba46-ba4ae8171ca9" (UID: "280b7c38-6ad1-4e93-ba46-ba4ae8171ca9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.731328 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.729858    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istiod-ca-cert" pod "061e2d92-1dd6-4373-b93a-09dd58d23a0e" (UID: "061e2d92-1dd6-4373-b93a-09dd58d23a0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:11.731328 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.730063    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-credential-socket" pod "280b7c38-6ad1-4e93-ba46-ba4ae8171ca9" (UID: "280b7c38-6ad1-4e93-ba46-ba4ae8171ca9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.731328 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.730295    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f891215-2cc6-4b56-8ba2-68687e58f21c-istio-token" pod "6f891215-2cc6-4b56-8ba2-68687e58f21c" (UID: "6f891215-2cc6-4b56-8ba2-68687e58f21c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:11.731328 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.730460    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-kube-api-access-d8bss" pod "280b7c38-6ad1-4e93-ba46-ba4ae8171ca9" (UID: "280b7c38-6ad1-4e93-ba46-ba4ae8171ca9"). InnerVolumeSpecName "kube-api-access-d8bss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:11.731328 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.730571    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f891215-2cc6-4b56-8ba2-68687e58f21c-kube-api-access-8zlwm" pod "6f891215-2cc6-4b56-8ba2-68687e58f21c" (UID: "6f891215-2cc6-4b56-8ba2-68687e58f21c"). InnerVolumeSpecName "kube-api-access-8zlwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:11.731328 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.730685    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-workload-socket" pod "280b7c38-6ad1-4e93-ba46-ba4ae8171ca9" (UID: "280b7c38-6ad1-4e93-ba46-ba4ae8171ca9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.731328 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.730735    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f891215-2cc6-4b56-8ba2-68687e58f21c-istiod-ca-cert" pod "6f891215-2cc6-4b56-8ba2-68687e58f21c" (UID: "6f891215-2cc6-4b56-8ba2-68687e58f21c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:11.731959 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.730793    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-credential-socket" pod "6f891215-2cc6-4b56-8ba2-68687e58f21c" (UID: "6f891215-2cc6-4b56-8ba2-68687e58f21c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.732510 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.732487    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f891215-2cc6-4b56-8ba2-68687e58f21c-istio-podinfo" pod "6f891215-2cc6-4b56-8ba2-68687e58f21c" (UID: "6f891215-2cc6-4b56-8ba2-68687e58f21c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:07:11.732904 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.732863    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061e2d92-1dd6-4373-b93a-09dd58d23a0e-kube-api-access-j8m2v" pod "061e2d92-1dd6-4373-b93a-09dd58d23a0e" (UID: "061e2d92-1dd6-4373-b93a-09dd58d23a0e"). InnerVolumeSpecName "kube-api-access-j8m2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:11.732996 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.732971    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istio-envoy" pod "061e2d92-1dd6-4373-b93a-09dd58d23a0e" (UID: "061e2d92-1dd6-4373-b93a-09dd58d23a0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.733071 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.733046    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istio-envoy" pod "280b7c38-6ad1-4e93-ba46-ba4ae8171ca9" (UID: "280b7c38-6ad1-4e93-ba46-ba4ae8171ca9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.733716 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.733692    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istio-token" pod "280b7c38-6ad1-4e93-ba46-ba4ae8171ca9" (UID: "280b7c38-6ad1-4e93-ba46-ba4ae8171ca9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:11.733829 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.733809    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istio-podinfo" pod "061e2d92-1dd6-4373-b93a-09dd58d23a0e" (UID: "061e2d92-1dd6-4373-b93a-09dd58d23a0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:07:11.734432 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.734409    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istio-token" pod "061e2d92-1dd6-4373-b93a-09dd58d23a0e" (UID: "061e2d92-1dd6-4373-b93a-09dd58d23a0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:11.734519 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.734437    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istio-podinfo" pod "280b7c38-6ad1-4e93-ba46-ba4ae8171ca9" (UID: "280b7c38-6ad1-4e93-ba46-ba4ae8171ca9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:07:11.758710 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.758684    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-istio-envoy" pod "6f891215-2cc6-4b56-8ba2-68687e58f21c" (UID: "6f891215-2cc6-4b56-8ba2-68687e58f21c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.830247 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.830218    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62fa8024-dac6-4993-b671-23bda513f309-kube-api-access-p2lkq\" (UniqueName: \"kubernetes.io/projected/62fa8024-dac6-4993-b671-23bda513f309-kube-api-access-p2lkq\") pod \"62fa8024-dac6-4993-b671-23bda513f309\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") "
Oct 30 22:07:11.830431 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.830264    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc565097-2279-4f89-a2a6-35e117fd929a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc565097-2279-4f89-a2a6-35e117fd929a-istio-podinfo\") pod \"bc565097-2279-4f89-a2a6-35e117fd929a\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") "
Oct 30 22:07:11.830431 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.830293    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62fa8024-dac6-4993-b671-23bda513f309-istio-token\" (UniqueName: \"kubernetes.io/projected/62fa8024-dac6-4993-b671-23bda513f309-istio-token\") pod \"62fa8024-dac6-4993-b671-23bda513f309\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") "
Oct 30 22:07:11.830431 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.830320    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-istio-data\") pod \"bc565097-2279-4f89-a2a6-35e117fd929a\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") "
Oct 30 22:07:11.830431 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.830384    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc565097-2279-4f89-a2a6-35e117fd929a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc565097-2279-4f89-a2a6-35e117fd929a-istiod-ca-cert\") pod \"bc565097-2279-4f89-a2a6-35e117fd929a\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") "
Oct 30 22:07:11.830431 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.830410    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc565097-2279-4f89-a2a6-35e117fd929a-kube-api-access-ljbv6\" (UniqueName: \"kubernetes.io/projected/bc565097-2279-4f89-a2a6-35e117fd929a-kube-api-access-ljbv6\") pod \"bc565097-2279-4f89-a2a6-35e117fd929a\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") "
Oct 30 22:07:11.831046 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.830469    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-istio-envoy\") pod \"bc565097-2279-4f89-a2a6-35e117fd929a\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") "
Oct 30 22:07:11.831046 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.830505    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-istio-envoy\") pod \"62fa8024-dac6-4993-b671-23bda513f309\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") "
Oct 30 22:07:11.831046 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.830552    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62fa8024-dac6-4993-b671-23bda513f309-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62fa8024-dac6-4993-b671-23bda513f309-istiod-ca-cert\") pod \"62fa8024-dac6-4993-b671-23bda513f309\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") "
Oct 30 22:07:11.831046 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.830588    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-workload-socket\") pod \"bc565097-2279-4f89-a2a6-35e117fd929a\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") "
Oct 30 22:07:11.831046 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.830621    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-credential-socket\") pod \"62fa8024-dac6-4993-b671-23bda513f309\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") "
Oct 30 22:07:11.831046 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.830651    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-istio-data\") pod \"62fa8024-dac6-4993-b671-23bda513f309\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") "
Oct 30 22:07:11.831046 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.830731    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-credential-socket\") pod \"bc565097-2279-4f89-a2a6-35e117fd929a\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") "
Oct 30 22:07:11.831046 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.830770    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc565097-2279-4f89-a2a6-35e117fd929a-istio-token\" (UniqueName: \"kubernetes.io/projected/bc565097-2279-4f89-a2a6-35e117fd929a-istio-token\") pod \"bc565097-2279-4f89-a2a6-35e117fd929a\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") "
Oct 30 22:07:11.831046 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.830834    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-workload-certs\") pod \"bc565097-2279-4f89-a2a6-35e117fd929a\" (UID: \"bc565097-2279-4f89-a2a6-35e117fd929a\") "
Oct 30 22:07:11.831046 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.830893    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-workload-certs\") pod \"62fa8024-dac6-4993-b671-23bda513f309\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") "
Oct 30 22:07:11.831046 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.830926    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-workload-socket\") pod \"62fa8024-dac6-4993-b671-23bda513f309\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") "
Oct 30 22:07:11.831046 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.830963    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62fa8024-dac6-4993-b671-23bda513f309-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62fa8024-dac6-4993-b671-23bda513f309-istio-podinfo\") pod \"62fa8024-dac6-4993-b671-23bda513f309\" (UID: \"62fa8024-dac6-4993-b671-23bda513f309\") "
Oct 30 22:07:11.832915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.831121    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-istio-data" pod "62fa8024-dac6-4993-b671-23bda513f309" (UID: "62fa8024-dac6-4993-b671-23bda513f309"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.832915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.831475    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-workload-certs" pod "62fa8024-dac6-4993-b671-23bda513f309" (UID: "62fa8024-dac6-4993-b671-23bda513f309"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.832915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.831607    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-workload-socket" pod "bc565097-2279-4f89-a2a6-35e117fd929a" (UID: "bc565097-2279-4f89-a2a6-35e117fd929a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.832915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.831708    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62fa8024-dac6-4993-b671-23bda513f309-istiod-ca-cert" pod "62fa8024-dac6-4993-b671-23bda513f309" (UID: "62fa8024-dac6-4993-b671-23bda513f309"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:11.832915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.831723    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-credential-socket" pod "bc565097-2279-4f89-a2a6-35e117fd929a" (UID: "bc565097-2279-4f89-a2a6-35e117fd929a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.832915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.831748    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-workload-socket" pod "62fa8024-dac6-4993-b671-23bda513f309" (UID: "62fa8024-dac6-4993-b671-23bda513f309"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.832915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.831202    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc565097-2279-4f89-a2a6-35e117fd929a-istiod-ca-cert" pod "bc565097-2279-4f89-a2a6-35e117fd929a" (UID: "bc565097-2279-4f89-a2a6-35e117fd929a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:11.832915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.831897    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-istio-data" pod "bc565097-2279-4f89-a2a6-35e117fd929a" (UID: "bc565097-2279-4f89-a2a6-35e117fd929a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.832915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.832034    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-credential-socket" pod "62fa8024-dac6-4993-b671-23bda513f309" (UID: "62fa8024-dac6-4993-b671-23bda513f309"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.832915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.832114    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-workload-certs" pod "bc565097-2279-4f89-a2a6-35e117fd929a" (UID: "bc565097-2279-4f89-a2a6-35e117fd929a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.832915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.832674    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.832915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.832700    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.832915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.832715    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.832915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.832730    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.832915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.832743    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.832915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.832756    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f891215-2cc6-4b56-8ba2-68687e58f21c-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.832915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.832773    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.832915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.832788    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.832915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.832803    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.832915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.832817    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.832915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.832831    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f891215-2cc6-4b56-8ba2-68687e58f21c-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.832915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.832844    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.832915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.832858    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.832915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.832871    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.832915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.832885    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8m2v\" (UniqueName: \"kubernetes.io/projected/061e2d92-1dd6-4373-b93a-09dd58d23a0e-kube-api-access-j8m2v\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.832915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.832898    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f891215-2cc6-4b56-8ba2-68687e58f21c-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.832915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.832912    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.832915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.832924    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/061e2d92-1dd6-4373-b93a-09dd58d23a0e-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.832915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.832943    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/061e2d92-1dd6-4373-b93a-09dd58d23a0e-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.834711 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.832958    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8bss\" (UniqueName: \"kubernetes.io/projected/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-kube-api-access-d8bss\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.834711 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.832974    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zlwm\" (UniqueName: \"kubernetes.io/projected/6f891215-2cc6-4b56-8ba2-68687e58f21c-kube-api-access-8zlwm\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.834711 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.832988    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.834711 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.833001    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62fa8024-dac6-4993-b671-23bda513f309-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.834711 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.833013    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.834711 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.833026    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.834711 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.833040    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.834711 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.833055    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f891215-2cc6-4b56-8ba2-68687e58f21c-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.835102 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.834877    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc565097-2279-4f89-a2a6-35e117fd929a-kube-api-access-ljbv6" pod "bc565097-2279-4f89-a2a6-35e117fd929a" (UID: "bc565097-2279-4f89-a2a6-35e117fd929a"). InnerVolumeSpecName "kube-api-access-ljbv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:11.835102 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.834988    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc565097-2279-4f89-a2a6-35e117fd929a-istio-podinfo" pod "bc565097-2279-4f89-a2a6-35e117fd929a" (UID: "bc565097-2279-4f89-a2a6-35e117fd929a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:07:11.835617 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.835520    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-istio-envoy" pod "bc565097-2279-4f89-a2a6-35e117fd929a" (UID: "bc565097-2279-4f89-a2a6-35e117fd929a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.835617 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.835527    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62fa8024-dac6-4993-b671-23bda513f309-kube-api-access-p2lkq" pod "62fa8024-dac6-4993-b671-23bda513f309" (UID: "62fa8024-dac6-4993-b671-23bda513f309"). InnerVolumeSpecName "kube-api-access-p2lkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:11.835617 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.835529    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-istio-envoy" pod "62fa8024-dac6-4993-b671-23bda513f309" (UID: "62fa8024-dac6-4993-b671-23bda513f309"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:11.836219 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.836187    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62fa8024-dac6-4993-b671-23bda513f309-istio-podinfo" pod "62fa8024-dac6-4993-b671-23bda513f309" (UID: "62fa8024-dac6-4993-b671-23bda513f309"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:07:11.836481 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.836457    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc565097-2279-4f89-a2a6-35e117fd929a-istio-token" pod "bc565097-2279-4f89-a2a6-35e117fd929a" (UID: "bc565097-2279-4f89-a2a6-35e117fd929a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:11.837079 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.837059    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62fa8024-dac6-4993-b671-23bda513f309-istio-token" pod "62fa8024-dac6-4993-b671-23bda513f309" (UID: "62fa8024-dac6-4993-b671-23bda513f309"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:11.934811 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.934777    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.934811 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.934808    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fa8024-dac6-4993-b671-23bda513f309-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.934811 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.934819    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc565097-2279-4f89-a2a6-35e117fd929a-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.935122 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.934828    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.935122 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.934836    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62fa8024-dac6-4993-b671-23bda513f309-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.935122 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.934845    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2lkq\" (UniqueName: \"kubernetes.io/projected/62fa8024-dac6-4993-b671-23bda513f309-kube-api-access-p2lkq\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.935122 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.934853    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc565097-2279-4f89-a2a6-35e117fd929a-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.935122 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.934861    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62fa8024-dac6-4993-b671-23bda513f309-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.935122 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.934870    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.935122 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.934879    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc565097-2279-4f89-a2a6-35e117fd929a-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.935122 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.934888    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljbv6\" (UniqueName: \"kubernetes.io/projected/bc565097-2279-4f89-a2a6-35e117fd929a-kube-api-access-ljbv6\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:11.935122 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:11.934895    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc565097-2279-4f89-a2a6-35e117fd929a-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:12.502105 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:12.502075    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"
Oct 30 22:07:12.502672 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:12.502075    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd" event={"ID":"280b7c38-6ad1-4e93-ba46-ba4ae8171ca9","Type":"ContainerDied","Data":"6560fdfb1806473855ae1c1cc27c6482c28a7bf80297aaa95e076c5cabfeedc1"}
Oct 30 22:07:12.502672 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:12.502270    5087 scope.go:122] "RemoveContainer" containerID="fb3deb51b4b87b4e24ff09d3aab21c372977abc28aec7479efca379e39630308"
Oct 30 22:07:12.504811 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:12.504382    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc" event={"ID":"322d51a7-324e-4f33-a606-3384bb72d3cc","Type":"ContainerDied","Data":"826b9963729dfbbc5b983f507b5073017cab3c63ea57ffdbae54b3fa27fe1686"}
Oct 30 22:07:12.504811 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:12.504481    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"
Oct 30 22:07:12.506474 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:12.506431    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"
Oct 30 22:07:12.506693 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:12.506484    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl" event={"ID":"061e2d92-1dd6-4373-b93a-09dd58d23a0e","Type":"ContainerDied","Data":"c926c7eec59b91be0a776214529426ca856ee771e279885f4629f93454766eab"}
Oct 30 22:07:12.509299 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:12.509240    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"
Oct 30 22:07:12.509621 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:12.509238    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk" event={"ID":"6f891215-2cc6-4b56-8ba2-68687e58f21c","Type":"ContainerDied","Data":"910a74cea5d2d0934daf4ae55946b23e184c8d84ff330aa16d4df4e31c2d02ed"}
Oct 30 22:07:12.526662 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:12.524416    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc" event={"ID":"62fa8024-dac6-4993-b671-23bda513f309","Type":"ContainerDied","Data":"0ebc3d1369780cba4d624673c9405f5e7eb8fcee4e2b987e18a100db0c5e6837"}
Oct 30 22:07:12.526662 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:12.524553    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"
Oct 30 22:07:12.528882 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:12.528849    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv" event={"ID":"bc565097-2279-4f89-a2a6-35e117fd929a","Type":"ContainerDied","Data":"5b7265d793a30a2c0fe17ba06997df5d08f2547feb6a6b4464941be571353285"}
Oct 30 22:07:12.529369 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:12.529290    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"
Oct 30 22:07:12.531666 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:12.531644    5087 scope.go:122] "RemoveContainer" containerID="083569d08e187f3975f006cc2162191cc791ca68796421440dd8ff3e28df75cf"
Oct 30 22:07:12.549042 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:12.548922    5087 scope.go:122] "RemoveContainer" containerID="94e5dfb2d66b0be6a057524b68d8131c4521f2cc668683d3cdcc993e11ff2650"
Oct 30 22:07:12.561956 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:12.561931    5087 scope.go:122] "RemoveContainer" containerID="d5bc6fca893b137116b81e4aa445275f19416b0b9faad4743e2ca6d3b960bebb"
Oct 30 22:07:12.575341 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:12.575320    5087 scope.go:122] "RemoveContainer" containerID="be67215a5cbffa912d1a3de9277cf14d96e9435b5c1ebe523d7461be348c7bcb"
Oct 30 22:07:12.589681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:12.589655    5087 scope.go:122] "RemoveContainer" containerID="3cc130c3ff474265a6b8dd8c8353b5ba573d5d7ed8469e2b235d7b9ba8beefa9"
Oct 30 22:07:12.667524 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:12.667488    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"]
Oct 30 22:07:12.680006 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:12.679972    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-d5bcl"]
Oct 30 22:07:12.788745 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:12.788662    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"]
Oct 30 22:07:12.793483 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:12.793457    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-5b9gk"]
Oct 30 22:07:12.900781 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:12.900753    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"]
Oct 30 22:07:12.906919 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:12.906894    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-hfdvc"]
Oct 30 22:07:13.015858 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:13.015816    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"]
Oct 30 22:07:13.020008 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:13.019983    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-jj2hd"]
Oct 30 22:07:13.129413 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:13.129317    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"]
Oct 30 22:07:13.134983 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:13.134958    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-fskjv"]
Oct 30 22:07:13.257373 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:13.256242    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"]
Oct 30 22:07:13.257373 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:13.256301    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-xmvjc"]
Oct 30 22:07:13.709820 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:13.709786    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="061e2d92-1dd6-4373-b93a-09dd58d23a0e" path="/var/lib/kubelet/pods/061e2d92-1dd6-4373-b93a-09dd58d23a0e/volumes"
Oct 30 22:07:13.710485 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:13.710464    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="280b7c38-6ad1-4e93-ba46-ba4ae8171ca9" path="/var/lib/kubelet/pods/280b7c38-6ad1-4e93-ba46-ba4ae8171ca9/volumes"
Oct 30 22:07:13.710985 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:13.710965    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="322d51a7-324e-4f33-a606-3384bb72d3cc" path="/var/lib/kubelet/pods/322d51a7-324e-4f33-a606-3384bb72d3cc/volumes"
Oct 30 22:07:13.711793 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:13.711774    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62fa8024-dac6-4993-b671-23bda513f309" path="/var/lib/kubelet/pods/62fa8024-dac6-4993-b671-23bda513f309/volumes"
Oct 30 22:07:13.712248 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:13.712230    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f891215-2cc6-4b56-8ba2-68687e58f21c" path="/var/lib/kubelet/pods/6f891215-2cc6-4b56-8ba2-68687e58f21c/volumes"
Oct 30 22:07:13.712730 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:13.712712    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc565097-2279-4f89-a2a6-35e117fd929a" path="/var/lib/kubelet/pods/bc565097-2279-4f89-a2a6-35e117fd929a/volumes"
Oct 30 22:07:15.426915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:15.426888    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78cf83-796db64fff-7ph2p"
Oct 30 22:07:17.000148 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.000119    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt"]
Oct 30 22:07:17.000683 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.000668    5087 cpu_manager.go:414] "containerMap: removing container" podUID="bc565097-2279-4f89-a2a6-35e117fd929a" containerName="istio-proxy"
Oct 30 22:07:17.000723 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.000683    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc565097-2279-4f89-a2a6-35e117fd929a" containerName="istio-proxy"
Oct 30 22:07:17.000723 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.000700    5087 cpu_manager.go:414] "containerMap: removing container" podUID="280b7c38-6ad1-4e93-ba46-ba4ae8171ca9" containerName="istio-proxy"
Oct 30 22:07:17.000723 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.000705    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="280b7c38-6ad1-4e93-ba46-ba4ae8171ca9" containerName="istio-proxy"
Oct 30 22:07:17.000817 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.000724    5087 cpu_manager.go:414] "containerMap: removing container" podUID="b2fda28b-f446-4917-bbe1-721ecff56fd8" containerName="istio-proxy"
Oct 30 22:07:17.000817 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.000730    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2fda28b-f446-4917-bbe1-721ecff56fd8" containerName="istio-proxy"
Oct 30 22:07:17.000817 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.000744    5087 cpu_manager.go:414] "containerMap: removing container" podUID="6f891215-2cc6-4b56-8ba2-68687e58f21c" containerName="istio-proxy"
Oct 30 22:07:17.000817 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.000749    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f891215-2cc6-4b56-8ba2-68687e58f21c" containerName="istio-proxy"
Oct 30 22:07:17.000817 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.000757    5087 cpu_manager.go:414] "containerMap: removing container" podUID="061e2d92-1dd6-4373-b93a-09dd58d23a0e" containerName="istio-proxy"
Oct 30 22:07:17.000817 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.000761    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="061e2d92-1dd6-4373-b93a-09dd58d23a0e" containerName="istio-proxy"
Oct 30 22:07:17.000817 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.000771    5087 cpu_manager.go:414] "containerMap: removing container" podUID="62fa8024-dac6-4993-b671-23bda513f309" containerName="istio-proxy"
Oct 30 22:07:17.000817 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.000775    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62fa8024-dac6-4993-b671-23bda513f309" containerName="istio-proxy"
Oct 30 22:07:17.000817 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.000784    5087 cpu_manager.go:414] "containerMap: removing container" podUID="322d51a7-324e-4f33-a606-3384bb72d3cc" containerName="istio-proxy"
Oct 30 22:07:17.000817 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.000789    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="322d51a7-324e-4f33-a606-3384bb72d3cc" containerName="istio-proxy"
Oct 30 22:07:17.000817 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.000797    5087 cpu_manager.go:414] "containerMap: removing container" podUID="66ef40b8-03ce-449b-ab82-da0b7af9e155" containerName="istio-proxy"
Oct 30 22:07:17.000817 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.000801    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66ef40b8-03ce-449b-ab82-da0b7af9e155" containerName="istio-proxy"
Oct 30 22:07:17.001144 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.000880    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="280b7c38-6ad1-4e93-ba46-ba4ae8171ca9" containerName="istio-proxy"
Oct 30 22:07:17.001144 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.000892    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2fda28b-f446-4917-bbe1-721ecff56fd8" containerName="istio-proxy"
Oct 30 22:07:17.001144 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.000899    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f891215-2cc6-4b56-8ba2-68687e58f21c" containerName="istio-proxy"
Oct 30 22:07:17.001144 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.000906    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="061e2d92-1dd6-4373-b93a-09dd58d23a0e" containerName="istio-proxy"
Oct 30 22:07:17.001144 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.000911    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="62fa8024-dac6-4993-b671-23bda513f309" containerName="istio-proxy"
Oct 30 22:07:17.001144 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.000917    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="322d51a7-324e-4f33-a606-3384bb72d3cc" containerName="istio-proxy"
Oct 30 22:07:17.001144 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.000922    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="66ef40b8-03ce-449b-ab82-da0b7af9e155" containerName="istio-proxy"
Oct 30 22:07:17.001144 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.000929    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc565097-2279-4f89-a2a6-35e117fd929a" containerName="istio-proxy"
Oct 30 22:07:17.004175 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.004159    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt"
Oct 30 22:07:17.107262 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.107229    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 22:07:17.107262 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.107258    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fqgfc\""
Oct 30 22:07:17.107546 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.107303    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-1db768-serving-cert\""
Oct 30 22:07:17.107922 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.107897    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae946a0-125e-4832-96eb-8b824f689a88-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt\" (UID: \"cae946a0-125e-4832-96eb-8b824f689a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt"
Oct 30 22:07:17.108090 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.108066    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae946a0-125e-4832-96eb-8b824f689a88-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt\" (UID: \"cae946a0-125e-4832-96eb-8b824f689a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt"
Oct 30 22:07:17.108144 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.108129    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvp9w\" (UniqueName: \"kubernetes.io/projected/cae946a0-125e-4832-96eb-8b824f689a88-kube-api-access-pvp9w\") pod \"maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt\" (UID: \"cae946a0-125e-4832-96eb-8b824f689a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt"
Oct 30 22:07:17.108201 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.108169    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae946a0-125e-4832-96eb-8b824f689a88-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt\" (UID: \"cae946a0-125e-4832-96eb-8b824f689a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt"
Oct 30 22:07:17.110391 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.110372    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt"]
Oct 30 22:07:17.185386 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:07:17.185319    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:07:17.209019 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.208990    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae946a0-125e-4832-96eb-8b824f689a88-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt\" (UID: \"cae946a0-125e-4832-96eb-8b824f689a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt"
Oct 30 22:07:17.209139 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.209095    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae946a0-125e-4832-96eb-8b824f689a88-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt\" (UID: \"cae946a0-125e-4832-96eb-8b824f689a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt"
Oct 30 22:07:17.209139 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.209116    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvp9w\" (UniqueName: \"kubernetes.io/projected/cae946a0-125e-4832-96eb-8b824f689a88-kube-api-access-pvp9w\") pod \"maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt\" (UID: \"cae946a0-125e-4832-96eb-8b824f689a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt"
Oct 30 22:07:17.209226 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.209149    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae946a0-125e-4832-96eb-8b824f689a88-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt\" (UID: \"cae946a0-125e-4832-96eb-8b824f689a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt"
Oct 30 22:07:17.209358 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:07:17.209323    5087 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-serving-cert: secret "maas-api-e2e-shared-b-w6-1db768-serving-cert" not found
Oct 30 22:07:17.209433 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:07:17.209421    5087 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cae946a0-125e-4832-96eb-8b824f689a88-maas-api-tls podName:cae946a0-125e-4832-96eb-8b824f689a88 nodeName:}" failed. No retries permitted until 2026-10-30 22:07:17.709396059 +0000 UTC m=+2032.183900033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cae946a0-125e-4832-96eb-8b824f689a88-maas-api-tls") pod "maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt" (UID: "cae946a0-125e-4832-96eb-8b824f689a88") : secret "maas-api-e2e-shared-b-w6-1db768-serving-cert" not found
Oct 30 22:07:17.209845 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.209822    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae946a0-125e-4832-96eb-8b824f689a88-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt\" (UID: \"cae946a0-125e-4832-96eb-8b824f689a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt"
Oct 30 22:07:17.211750 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.211732    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae946a0-125e-4832-96eb-8b824f689a88-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt\" (UID: \"cae946a0-125e-4832-96eb-8b824f689a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt"
Oct 30 22:07:17.267856 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.267727    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvp9w\" (UniqueName: \"kubernetes.io/projected/cae946a0-125e-4832-96eb-8b824f689a88-kube-api-access-pvp9w\") pod \"maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt\" (UID: \"cae946a0-125e-4832-96eb-8b824f689a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt"
Oct 30 22:07:17.716022 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.715943    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae946a0-125e-4832-96eb-8b824f689a88-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt\" (UID: \"cae946a0-125e-4832-96eb-8b824f689a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt"
Oct 30 22:07:17.718540 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.718516    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae946a0-125e-4832-96eb-8b824f689a88-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt\" (UID: \"cae946a0-125e-4832-96eb-8b824f689a88\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt"
Oct 30 22:07:17.918079 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:17.918039    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt"
Oct 30 22:07:18.298463 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:18.298433    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt"]
Oct 30 22:07:18.300247 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:07:18.300218    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcae946a0_125e_4832_96eb_8b824f689a88.slice/crio-abc4555092f92ff51b0a907a8daadeab113c1f1d0ab15c2d0c2361bd04e3e397 WatchSource:0}: Error finding container abc4555092f92ff51b0a907a8daadeab113c1f1d0ab15c2d0c2361bd04e3e397: Status 404 returned error can't find the container with id abc4555092f92ff51b0a907a8daadeab113c1f1d0ab15c2d0c2361bd04e3e397
Oct 30 22:07:18.301899 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:18.301878    5087 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:07:18.617132 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:18.617049    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt" event={"ID":"cae946a0-125e-4832-96eb-8b824f689a88","Type":"ContainerStarted","Data":"abc4555092f92ff51b0a907a8daadeab113c1f1d0ab15c2d0c2361bd04e3e397"}
Oct 30 22:07:19.428652 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.428615    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"]
Oct 30 22:07:19.435138 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.434965    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:19.497429 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.488686    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-307368-openshift-default-dockercfg-xjc2t\""
Oct 30 22:07:19.584477 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.569110    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/164e63e1-b309-4236-983a-2fd171af5b14-istiod-ca-cert\") pod \"e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") " pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:19.584477 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.569177    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-workload-socket\") pod \"e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") " pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:19.584477 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.573400    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/164e63e1-b309-4236-983a-2fd171af5b14-istio-podinfo\") pod \"e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") " pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:19.584477 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.573704    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9l5v\" (UniqueName: \"kubernetes.io/projected/164e63e1-b309-4236-983a-2fd171af5b14-kube-api-access-v9l5v\") pod \"e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") " pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:19.584477 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.573814    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-credential-socket\") pod \"e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") " pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:19.584477 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.573871    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-istio-data\") pod \"e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") " pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:19.584941 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.584424    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-workload-certs\") pod \"e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") " pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:19.584941 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.584810    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-istio-envoy\") pod \"e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") " pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:19.584941 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.584918    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/164e63e1-b309-4236-983a-2fd171af5b14-istio-token\") pod \"e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") " pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:19.658136 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.658102    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"]
Oct 30 22:07:19.736373 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.734777    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9l5v\" (UniqueName: \"kubernetes.io/projected/164e63e1-b309-4236-983a-2fd171af5b14-kube-api-access-v9l5v\") pod \"e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") " pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:19.736373 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.734889    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-credential-socket\") pod \"e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") " pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:19.736373 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.734930    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-istio-data\") pod \"e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") " pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:19.736373 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.734965    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-workload-certs\") pod \"e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") " pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:19.736373 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.735013    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-istio-envoy\") pod \"e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") " pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:19.736373 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.735079    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/164e63e1-b309-4236-983a-2fd171af5b14-istio-token\") pod \"e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") " pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:19.736373 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.735126    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/164e63e1-b309-4236-983a-2fd171af5b14-istiod-ca-cert\") pod \"e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") " pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:19.736373 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.735155    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-workload-socket\") pod \"e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") " pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:19.736373 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.735195    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/164e63e1-b309-4236-983a-2fd171af5b14-istio-podinfo\") pod \"e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") " pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:19.736373 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.735552    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-istio-data\") pod \"e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") " pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:19.736373 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.736136    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-credential-socket\") pod \"e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") " pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:19.737098 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.736651    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-workload-certs\") pod \"e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") " pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:19.748675 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.748647    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-istio-envoy\") pod \"e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") " pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:19.750915 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.750885    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/164e63e1-b309-4236-983a-2fd171af5b14-istio-podinfo\") pod \"e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") " pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:19.751562 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.751537    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/164e63e1-b309-4236-983a-2fd171af5b14-istiod-ca-cert\") pod \"e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") " pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:19.751825 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.751784    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-workload-socket\") pod \"e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") " pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:19.802001 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.800891    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/164e63e1-b309-4236-983a-2fd171af5b14-istio-token\") pod \"e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") " pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:19.802001 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.801580    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9l5v\" (UniqueName: \"kubernetes.io/projected/164e63e1-b309-4236-983a-2fd171af5b14-kube-api-access-v9l5v\") pod \"e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") " pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:19.802001 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:19.801814    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:20.404987 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:20.404892    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"]
Oct 30 22:07:20.441262 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:20.441220    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:07:20.441829 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:20.441315    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:07:20.441829 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:20.441375    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:07:20.717839 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:20.715711    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz" event={"ID":"164e63e1-b309-4236-983a-2fd171af5b14","Type":"ContainerStarted","Data":"e1aeff8d7817ae5828f5734121dd9658c793153519f5374b39cc49105fd08bea"}
Oct 30 22:07:20.717839 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:20.715813    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz" event={"ID":"164e63e1-b309-4236-983a-2fd171af5b14","Type":"ContainerStarted","Data":"97709c773f8d361e3d7bb896c86ddcbdc84f84904f00dcf04135c50a8ae5035c"}
Oct 30 22:07:20.804399 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:20.802694    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:21.810625 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:21.810596    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:21.985834 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:21.985774    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz" podStartSLOduration=2.985756005 podStartE2EDuration="2.985756005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:20.836606955 +0000 UTC m=+2035.311110936" watchObservedRunningTime="2026-10-30 22:07:21.985756005 +0000 UTC m=+2036.460259984"
Oct 30 22:07:22.740371 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:22.740320    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt" event={"ID":"cae946a0-125e-4832-96eb-8b824f689a88","Type":"ContainerStarted","Data":"2eb96bdd43dcbf0bf9cc88348babd0a34803f9262386e065df30df80637bbbdb"}
Oct 30 22:07:22.740581 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:22.740419    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt"
Oct 30 22:07:22.740838 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:22.740818    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:22.741965 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:22.741942    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:07:22.857902 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:22.857848    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt" podStartSLOduration=2.808266967 podStartE2EDuration="6.857834464s" totalImagesPullingTime="4.049567497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:22.85549884 +0000 UTC m=+2037.330002819" watchObservedRunningTime="2026-10-30 22:07:22.857834464 +0000 UTC m=+2037.332338444"
Oct 30 22:07:27.964401 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:27.959900    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-307368-68678879f5-dtgqs"]
Oct 30 22:07:27.965026 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:27.965005    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-307368-68678879f5-dtgqs"
Oct 30 22:07:27.970968 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:27.970926    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c96595e9-95b1-4a92-a7ad-93932a2001ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-307368-68678879f5-dtgqs\" (UID: \"c96595e9-95b1-4a92-a7ad-93932a2001ba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-307368-68678879f5-dtgqs"
Oct 30 22:07:27.971143 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:27.971018    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbw9f\" (UniqueName: \"kubernetes.io/projected/c96595e9-95b1-4a92-a7ad-93932a2001ba-kube-api-access-xbw9f\") pod \"payload-pre-processing-e2e-api-b-307368-68678879f5-dtgqs\" (UID: \"c96595e9-95b1-4a92-a7ad-93932a2001ba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-307368-68678879f5-dtgqs"
Oct 30 22:07:28.017773 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:28.017734    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-307368\""
Oct 30 22:07:28.023425 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:28.023395    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-307368-dockercfg-d8l5h\""
Oct 30 22:07:28.023651 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:28.023526    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-307368-68678879f5-dtgqs"]
Oct 30 22:07:28.072698 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:28.072648    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c96595e9-95b1-4a92-a7ad-93932a2001ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-307368-68678879f5-dtgqs\" (UID: \"c96595e9-95b1-4a92-a7ad-93932a2001ba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-307368-68678879f5-dtgqs"
Oct 30 22:07:28.072895 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:28.072723    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbw9f\" (UniqueName: \"kubernetes.io/projected/c96595e9-95b1-4a92-a7ad-93932a2001ba-kube-api-access-xbw9f\") pod \"payload-pre-processing-e2e-api-b-307368-68678879f5-dtgqs\" (UID: \"c96595e9-95b1-4a92-a7ad-93932a2001ba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-307368-68678879f5-dtgqs"
Oct 30 22:07:28.073652 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:28.073625    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c96595e9-95b1-4a92-a7ad-93932a2001ba-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-307368-68678879f5-dtgqs\" (UID: \"c96595e9-95b1-4a92-a7ad-93932a2001ba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-307368-68678879f5-dtgqs"
Oct 30 22:07:28.103281 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:28.103243    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-307368-7574bb79f5-t8w22"]
Oct 30 22:07:28.107392 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:28.107334    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-307368-7574bb79f5-t8w22"
Oct 30 22:07:28.133803 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:28.133771    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbw9f\" (UniqueName: \"kubernetes.io/projected/c96595e9-95b1-4a92-a7ad-93932a2001ba-kube-api-access-xbw9f\") pod \"payload-pre-processing-e2e-api-b-307368-68678879f5-dtgqs\" (UID: \"c96595e9-95b1-4a92-a7ad-93932a2001ba\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-307368-68678879f5-dtgqs"
Oct 30 22:07:28.165335 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:28.165300    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-307368-7574bb79f5-t8w22"]
Oct 30 22:07:28.173888 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:28.173826    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb28650-f1aa-4f60-a062-bd7dc84418eb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-307368-7574bb79f5-t8w22\" (UID: \"dcb28650-f1aa-4f60-a062-bd7dc84418eb\") " pod="openshift-ingress/payload-processing-e2e-api-b-307368-7574bb79f5-t8w22"
Oct 30 22:07:28.174049 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:28.173939    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfw85\" (UniqueName: \"kubernetes.io/projected/dcb28650-f1aa-4f60-a062-bd7dc84418eb-kube-api-access-mfw85\") pod \"payload-processing-e2e-api-b-307368-7574bb79f5-t8w22\" (UID: \"dcb28650-f1aa-4f60-a062-bd7dc84418eb\") " pod="openshift-ingress/payload-processing-e2e-api-b-307368-7574bb79f5-t8w22"
Oct 30 22:07:28.278377 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:28.277374    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb28650-f1aa-4f60-a062-bd7dc84418eb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-307368-7574bb79f5-t8w22\" (UID: \"dcb28650-f1aa-4f60-a062-bd7dc84418eb\") " pod="openshift-ingress/payload-processing-e2e-api-b-307368-7574bb79f5-t8w22"
Oct 30 22:07:28.278377 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:28.277507    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfw85\" (UniqueName: \"kubernetes.io/projected/dcb28650-f1aa-4f60-a062-bd7dc84418eb-kube-api-access-mfw85\") pod \"payload-processing-e2e-api-b-307368-7574bb79f5-t8w22\" (UID: \"dcb28650-f1aa-4f60-a062-bd7dc84418eb\") " pod="openshift-ingress/payload-processing-e2e-api-b-307368-7574bb79f5-t8w22"
Oct 30 22:07:28.293932 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:28.293278    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb28650-f1aa-4f60-a062-bd7dc84418eb-plugins-config-volume\") pod \"payload-processing-e2e-api-b-307368-7574bb79f5-t8w22\" (UID: \"dcb28650-f1aa-4f60-a062-bd7dc84418eb\") " pod="openshift-ingress/payload-processing-e2e-api-b-307368-7574bb79f5-t8w22"
Oct 30 22:07:28.307410 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:28.302468    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-307368-68678879f5-dtgqs"
Oct 30 22:07:28.344041 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:28.336680    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfw85\" (UniqueName: \"kubernetes.io/projected/dcb28650-f1aa-4f60-a062-bd7dc84418eb-kube-api-access-mfw85\") pod \"payload-processing-e2e-api-b-307368-7574bb79f5-t8w22\" (UID: \"dcb28650-f1aa-4f60-a062-bd7dc84418eb\") " pod="openshift-ingress/payload-processing-e2e-api-b-307368-7574bb79f5-t8w22"
Oct 30 22:07:28.428792 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:28.422668    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-307368-7574bb79f5-t8w22"
Oct 30 22:07:28.746448 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:28.746419    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-307368-68678879f5-dtgqs"]
Oct 30 22:07:28.746945 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:07:28.746912    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc96595e9_95b1_4a92_a7ad_93932a2001ba.slice/crio-14685ad41d17bdf79fdbaf255ddc028c869c408ad88b1b86b18ebf1327b04ae3 WatchSource:0}: Error finding container 14685ad41d17bdf79fdbaf255ddc028c869c408ad88b1b86b18ebf1327b04ae3: Status 404 returned error can't find the container with id 14685ad41d17bdf79fdbaf255ddc028c869c408ad88b1b86b18ebf1327b04ae3
Oct 30 22:07:28.765755 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:28.752986    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt"
Oct 30 22:07:28.783059 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:28.783030    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-307368-68678879f5-dtgqs" event={"ID":"c96595e9-95b1-4a92-a7ad-93932a2001ba","Type":"ContainerStarted","Data":"14685ad41d17bdf79fdbaf255ddc028c869c408ad88b1b86b18ebf1327b04ae3"}
Oct 30 22:07:28.822842 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:28.822813    5087 scope.go:122] "RemoveContainer" containerID="786a37f9508f8b8ba28ab4d1eb8e091e161d9c5ce7239f1d3f130bc9b97e9c8b"
Oct 30 22:07:28.857402 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:28.857377    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-307368-7574bb79f5-t8w22"]
Oct 30 22:07:28.858591 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:07:28.858563    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcb28650_f1aa_4f60_a062_bd7dc84418eb.slice/crio-831668da4e94568b5d98347547ac880179435266b517713a58130b9e918ea8ee WatchSource:0}: Error finding container 831668da4e94568b5d98347547ac880179435266b517713a58130b9e918ea8ee: Status 404 returned error can't find the container with id 831668da4e94568b5d98347547ac880179435266b517713a58130b9e918ea8ee
Oct 30 22:07:29.792087 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:29.792043    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-307368-7574bb79f5-t8w22" event={"ID":"dcb28650-f1aa-4f60-a062-bd7dc84418eb","Type":"ContainerStarted","Data":"9442fd8b27cbf88cfce8755285a9248c0f23a451bf092adf71a5da647d3cc455"}
Oct 30 22:07:29.792566 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:29.792117    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-307368-7574bb79f5-t8w22"
Oct 30 22:07:29.792566 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:29.792142    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-307368-7574bb79f5-t8w22" event={"ID":"dcb28650-f1aa-4f60-a062-bd7dc84418eb","Type":"ContainerStarted","Data":"831668da4e94568b5d98347547ac880179435266b517713a58130b9e918ea8ee"}
Oct 30 22:07:29.793929 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:29.793902    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-307368-68678879f5-dtgqs" event={"ID":"c96595e9-95b1-4a92-a7ad-93932a2001ba","Type":"ContainerStarted","Data":"43d95aae1dff34f4c912fbfdd01082bb7dd090c756f39fa349d72fb6fcbeaf80"}
Oct 30 22:07:29.794055 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:29.793965    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-307368-68678879f5-dtgqs"
Oct 30 22:07:29.909525 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:29.909472    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-307368-7574bb79f5-t8w22" podStartSLOduration=1.909457317 podStartE2EDuration="1.909457317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:29.905972105 +0000 UTC m=+2044.380476112" watchObservedRunningTime="2026-10-30 22:07:29.909457317 +0000 UTC m=+2044.383961295"
Oct 30 22:07:30.023707 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:30.023628    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-307368-68678879f5-dtgqs" podStartSLOduration=3.023605485 podStartE2EDuration="3.023605485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:30.022631818 +0000 UTC m=+2044.497135798" watchObservedRunningTime="2026-10-30 22:07:30.023605485 +0000 UTC m=+2044.498109462"
Oct 30 22:07:40.804252 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:40.804224    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-307368-7574bb79f5-t8w22"
Oct 30 22:07:40.804683 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:40.804303    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-307368-68678879f5-dtgqs"
Oct 30 22:07:47.169589 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:07:47.169548    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:07:52.164796 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:52.164763    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-99f69c-5pvr7"]
Oct 30 22:07:52.168990 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:52.168973    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-99f69c-5pvr7"
Oct 30 22:07:52.220989 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:52.220955    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:07:52.270535 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:52.270506    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:07:52.273716 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:52.273691    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-99f69c-5pvr7"]
Oct 30 22:07:52.299694 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:52.299667    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkt9s\" (UniqueName: \"kubernetes.io/projected/f2f5bf30-e073-4b5a-be6f-48d0e07754d6-kube-api-access-mkt9s\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-99f69c-5pvr7\" (UID: \"f2f5bf30-e073-4b5a-be6f-48d0e07754d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-99f69c-5pvr7"
Oct 30 22:07:52.299895 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:52.299748    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2f5bf30-e073-4b5a-be6f-48d0e07754d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-99f69c-5pvr7\" (UID: \"f2f5bf30-e073-4b5a-be6f-48d0e07754d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-99f69c-5pvr7"
Oct 30 22:07:52.299971 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:52.299896    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2f5bf30-e073-4b5a-be6f-48d0e07754d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-99f69c-5pvr7\" (UID: \"f2f5bf30-e073-4b5a-be6f-48d0e07754d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-99f69c-5pvr7"
Oct 30 22:07:52.401597 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:52.401567    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkt9s\" (UniqueName: \"kubernetes.io/projected/f2f5bf30-e073-4b5a-be6f-48d0e07754d6-kube-api-access-mkt9s\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-99f69c-5pvr7\" (UID: \"f2f5bf30-e073-4b5a-be6f-48d0e07754d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-99f69c-5pvr7"
Oct 30 22:07:52.401745 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:52.401642    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2f5bf30-e073-4b5a-be6f-48d0e07754d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-99f69c-5pvr7\" (UID: \"f2f5bf30-e073-4b5a-be6f-48d0e07754d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-99f69c-5pvr7"
Oct 30 22:07:52.401745 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:52.401694    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2f5bf30-e073-4b5a-be6f-48d0e07754d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-99f69c-5pvr7\" (UID: \"f2f5bf30-e073-4b5a-be6f-48d0e07754d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-99f69c-5pvr7"
Oct 30 22:07:52.402304 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:52.402279    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2f5bf30-e073-4b5a-be6f-48d0e07754d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-99f69c-5pvr7\" (UID: \"f2f5bf30-e073-4b5a-be6f-48d0e07754d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-99f69c-5pvr7"
Oct 30 22:07:52.404121 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:52.404099    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2f5bf30-e073-4b5a-be6f-48d0e07754d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-99f69c-5pvr7\" (UID: \"f2f5bf30-e073-4b5a-be6f-48d0e07754d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-99f69c-5pvr7"
Oct 30 22:07:52.459005 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:52.458946    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkt9s\" (UniqueName: \"kubernetes.io/projected/f2f5bf30-e073-4b5a-be6f-48d0e07754d6-kube-api-access-mkt9s\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-99f69c-5pvr7\" (UID: \"f2f5bf30-e073-4b5a-be6f-48d0e07754d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-99f69c-5pvr7"
Oct 30 22:07:52.483511 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:52.483486    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-99f69c-5pvr7"
Oct 30 22:07:52.873497 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:52.873468    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-99f69c-5pvr7"]
Oct 30 22:07:52.875944 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:07:52.875906    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f5bf30_e073_4b5a_be6f_48d0e07754d6.slice/crio-1901764b504517926a101c3951d4a9f194fd25e9e4b173c1a8aa34e2a0f0518a WatchSource:0}: Error finding container 1901764b504517926a101c3951d4a9f194fd25e9e4b173c1a8aa34e2a0f0518a: Status 404 returned error can't find the container with id 1901764b504517926a101c3951d4a9f194fd25e9e4b173c1a8aa34e2a0f0518a
Oct 30 22:07:52.957965 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:52.957928    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-99f69c-5pvr7" event={"ID":"f2f5bf30-e073-4b5a-be6f-48d0e07754d6","Type":"ContainerStarted","Data":"1901764b504517926a101c3951d4a9f194fd25e9e4b173c1a8aa34e2a0f0518a"}
Oct 30 22:07:53.968747 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:53.968711    5087 generic.go:356] "Generic (PLEG): container finished" podID="f2f5bf30-e073-4b5a-be6f-48d0e07754d6" containerID="5cc2e7a3668b0e98a898b505c66c90fb34ecb782464f4c0a1ca8e80175d02e67" exitCode=0
Oct 30 22:07:53.969167 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:53.968801    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-99f69c-5pvr7" event={"ID":"f2f5bf30-e073-4b5a-be6f-48d0e07754d6","Type":"ContainerDied","Data":"5cc2e7a3668b0e98a898b505c66c90fb34ecb782464f4c0a1ca8e80175d02e67"}
Oct 30 22:07:55.146448 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:55.146417    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-99f69c-5pvr7"
Oct 30 22:07:55.244400 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:55.244329    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2f5bf30-e073-4b5a-be6f-48d0e07754d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2f5bf30-e073-4b5a-be6f-48d0e07754d6-maas-api-xks-ca\") pod \"f2f5bf30-e073-4b5a-be6f-48d0e07754d6\" (UID: \"f2f5bf30-e073-4b5a-be6f-48d0e07754d6\") "
Oct 30 22:07:55.244630 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:55.244502    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2f5bf30-e073-4b5a-be6f-48d0e07754d6-kube-api-access-mkt9s\" (UniqueName: \"kubernetes.io/projected/f2f5bf30-e073-4b5a-be6f-48d0e07754d6-kube-api-access-mkt9s\") pod \"f2f5bf30-e073-4b5a-be6f-48d0e07754d6\" (UID: \"f2f5bf30-e073-4b5a-be6f-48d0e07754d6\") "
Oct 30 22:07:55.244630 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:55.244594    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2f5bf30-e073-4b5a-be6f-48d0e07754d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2f5bf30-e073-4b5a-be6f-48d0e07754d6-maas-api-openshift-ca\") pod \"f2f5bf30-e073-4b5a-be6f-48d0e07754d6\" (UID: \"f2f5bf30-e073-4b5a-be6f-48d0e07754d6\") "
Oct 30 22:07:55.244969 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:55.244938    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2f5bf30-e073-4b5a-be6f-48d0e07754d6-maas-api-openshift-ca" pod "f2f5bf30-e073-4b5a-be6f-48d0e07754d6" (UID: "f2f5bf30-e073-4b5a-be6f-48d0e07754d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:55.245253 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:55.245234    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2f5bf30-e073-4b5a-be6f-48d0e07754d6-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:55.247402 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:55.247362    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2f5bf30-e073-4b5a-be6f-48d0e07754d6-maas-api-xks-ca" pod "f2f5bf30-e073-4b5a-be6f-48d0e07754d6" (UID: "f2f5bf30-e073-4b5a-be6f-48d0e07754d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:07:55.247402 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:55.247384    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2f5bf30-e073-4b5a-be6f-48d0e07754d6-kube-api-access-mkt9s" pod "f2f5bf30-e073-4b5a-be6f-48d0e07754d6" (UID: "f2f5bf30-e073-4b5a-be6f-48d0e07754d6"). InnerVolumeSpecName "kube-api-access-mkt9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:55.347090 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:55.347059    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2f5bf30-e073-4b5a-be6f-48d0e07754d6-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:55.347090 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:55.347089    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkt9s\" (UniqueName: \"kubernetes.io/projected/f2f5bf30-e073-4b5a-be6f-48d0e07754d6-kube-api-access-mkt9s\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:07:55.988204 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:55.988164    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-99f69c-5pvr7" event={"ID":"f2f5bf30-e073-4b5a-be6f-48d0e07754d6","Type":"ContainerDied","Data":"1901764b504517926a101c3951d4a9f194fd25e9e4b173c1a8aa34e2a0f0518a"}
Oct 30 22:07:55.988431 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:55.988209    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-99f69c-5pvr7"
Oct 30 22:07:55.988431 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:07:55.988229    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1901764b504517926a101c3951d4a9f194fd25e9e4b173c1a8aa34e2a0f0518a"
Oct 30 22:08:00.088486 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:00.088444    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7051-be6eab"]
Oct 30 22:08:00.089277 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:00.089253    5087 cpu_manager.go:414] "containerMap: removing container" podUID="f2f5bf30-e073-4b5a-be6f-48d0e07754d6" containerName="revoke-keys"
Oct 30 22:08:00.089406 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:00.089277    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2f5bf30-e073-4b5a-be6f-48d0e07754d6" containerName="revoke-keys"
Oct 30 22:08:00.089481 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:00.089454    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2f5bf30-e073-4b5a-be6f-48d0e07754d6" containerName="revoke-keys"
Oct 30 22:08:00.093255 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:00.093234    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7051-be6eab"
Oct 30 22:08:00.145881 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:00.145845    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7051-be6eab"]
Oct 30 22:08:00.146672 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:00.146645    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fs5vb\""
Oct 30 22:08:00.213134 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:00.213091    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnwsn\" (UniqueName: \"kubernetes.io/projected/bd2c99d8-4f03-454a-9483-f6670a8875bd-kube-api-access-bnwsn\") pod \"test-curl-7051-be6eab\" (UID: \"bd2c99d8-4f03-454a-9483-f6670a8875bd\") " pod="opendatahub/test-curl-7051-be6eab"
Oct 30 22:08:00.314855 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:00.314814    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnwsn\" (UniqueName: \"kubernetes.io/projected/bd2c99d8-4f03-454a-9483-f6670a8875bd-kube-api-access-bnwsn\") pod \"test-curl-7051-be6eab\" (UID: \"bd2c99d8-4f03-454a-9483-f6670a8875bd\") " pod="opendatahub/test-curl-7051-be6eab"
Oct 30 22:08:00.371315 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:00.371235    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnwsn\" (UniqueName: \"kubernetes.io/projected/bd2c99d8-4f03-454a-9483-f6670a8875bd-kube-api-access-bnwsn\") pod \"test-curl-7051-be6eab\" (UID: \"bd2c99d8-4f03-454a-9483-f6670a8875bd\") " pod="opendatahub/test-curl-7051-be6eab"
Oct 30 22:08:00.405513 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:00.405482    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7051-be6eab"
Oct 30 22:08:00.786826 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:00.786798    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7051-be6eab"]
Oct 30 22:08:00.787213 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:08:00.787187    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd2c99d8_4f03_454a_9483_f6670a8875bd.slice/crio-cd745d7a50a87eaa1d6cb310356f554e101366635f28118e3dac7dbb86efc2eb WatchSource:0}: Error finding container cd745d7a50a87eaa1d6cb310356f554e101366635f28118e3dac7dbb86efc2eb: Status 404 returned error can't find the container with id cd745d7a50a87eaa1d6cb310356f554e101366635f28118e3dac7dbb86efc2eb
Oct 30 22:08:01.022024 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:01.021984    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7051-be6eab" event={"ID":"bd2c99d8-4f03-454a-9483-f6670a8875bd","Type":"ContainerStarted","Data":"a1ca6bd9864b526ad0d6b13039294e87ddd791d5f375ce2ff0a9dc2519d7db5e"}
Oct 30 22:08:01.022189 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:01.022039    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7051-be6eab" event={"ID":"bd2c99d8-4f03-454a-9483-f6670a8875bd","Type":"ContainerStarted","Data":"cd745d7a50a87eaa1d6cb310356f554e101366635f28118e3dac7dbb86efc2eb"}
Oct 30 22:08:01.137866 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:01.137758    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7051-be6eab" podStartSLOduration=1.137742589 podStartE2EDuration="1.137742589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:01.134951718 +0000 UTC m=+2075.609455699" watchObservedRunningTime="2026-10-30 22:08:01.137742589 +0000 UTC m=+2075.612246568"
Oct 30 22:08:02.617695 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:02.617653    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7051-be6eab"]
Oct 30 22:08:02.621739 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:02.621701    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7051-be6eab"]
Oct 30 22:08:03.046429 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:03.046385    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7051-be6eab" podUID="bd2c99d8-4f03-454a-9483-f6670a8875bd" containerName="test-curl-7051-be6eab" containerID="cri-o://a1ca6bd9864b526ad0d6b13039294e87ddd791d5f375ce2ff0a9dc2519d7db5e" gracePeriod=2
Oct 30 22:08:03.099174 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:08:03.099138    5087 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7051-be6eab\" is forbidden: User \"system:node:ip-10-0-132-92.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-92.us-west-1.compute.internal' and this object" podUID="bd2c99d8-4f03-454a-9483-f6670a8875bd" pod="opendatahub/test-curl-7051-be6eab"
Oct 30 22:08:03.573813 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:03.573779    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr"]
Oct 30 22:08:03.574523 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:03.574504    5087 cpu_manager.go:414] "containerMap: removing container" podUID="bd2c99d8-4f03-454a-9483-f6670a8875bd" containerName="test-curl-7051-be6eab"
Oct 30 22:08:03.574657 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:03.574523    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd2c99d8-4f03-454a-9483-f6670a8875bd" containerName="test-curl-7051-be6eab"
Oct 30 22:08:03.574713 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:03.574661    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd2c99d8-4f03-454a-9483-f6670a8875bd" containerName="test-curl-7051-be6eab"
Oct 30 22:08:03.578292 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:03.578273    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr"
Oct 30 22:08:03.629034 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:08:03.629007    5087 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7051-be6eab\" is forbidden: User \"system:node:ip-10-0-132-92.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-92.us-west-1.compute.internal' and this object" podUID="bd2c99d8-4f03-454a-9483-f6670a8875bd" pod="opendatahub/test-curl-7051-be6eab"
Oct 30 22:08:03.630663 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:03.630639    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:08:03.635071 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:03.635049    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr"]
Oct 30 22:08:03.669656 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:03.669623    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18b3281c-0a59-4598-b4cc-b3a4565b0c01-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr\" (UID: \"18b3281c-0a59-4598-b4cc-b3a4565b0c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr"
Oct 30 22:08:03.669847 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:03.669825    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18b3281c-0a59-4598-b4cc-b3a4565b0c01-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr\" (UID: \"18b3281c-0a59-4598-b4cc-b3a4565b0c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr"
Oct 30 22:08:03.670066 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:03.670043    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn8qg\" (UniqueName: \"kubernetes.io/projected/18b3281c-0a59-4598-b4cc-b3a4565b0c01-kube-api-access-nn8qg\") pod \"maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr\" (UID: \"18b3281c-0a59-4598-b4cc-b3a4565b0c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr"
Oct 30 22:08:03.678513 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:03.678489    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:08:03.771049 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:03.771017    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn8qg\" (UniqueName: \"kubernetes.io/projected/18b3281c-0a59-4598-b4cc-b3a4565b0c01-kube-api-access-nn8qg\") pod \"maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr\" (UID: \"18b3281c-0a59-4598-b4cc-b3a4565b0c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr"
Oct 30 22:08:03.771241 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:03.771157    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18b3281c-0a59-4598-b4cc-b3a4565b0c01-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr\" (UID: \"18b3281c-0a59-4598-b4cc-b3a4565b0c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr"
Oct 30 22:08:03.771241 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:03.771197    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18b3281c-0a59-4598-b4cc-b3a4565b0c01-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr\" (UID: \"18b3281c-0a59-4598-b4cc-b3a4565b0c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr"
Oct 30 22:08:03.771939 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:03.771915    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18b3281c-0a59-4598-b4cc-b3a4565b0c01-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr\" (UID: \"18b3281c-0a59-4598-b4cc-b3a4565b0c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr"
Oct 30 22:08:03.773706 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:03.773688    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18b3281c-0a59-4598-b4cc-b3a4565b0c01-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr\" (UID: \"18b3281c-0a59-4598-b4cc-b3a4565b0c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr"
Oct 30 22:08:03.827968 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:03.827909    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn8qg\" (UniqueName: \"kubernetes.io/projected/18b3281c-0a59-4598-b4cc-b3a4565b0c01-kube-api-access-nn8qg\") pod \"maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr\" (UID: \"18b3281c-0a59-4598-b4cc-b3a4565b0c01\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr"
Oct 30 22:08:03.893596 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:03.893560    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr"
Oct 30 22:08:04.296993 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:04.296947    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr"]
Oct 30 22:08:05.070422 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:05.070389    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr" event={"ID":"18b3281c-0a59-4598-b4cc-b3a4565b0c01","Type":"ContainerStarted","Data":"91fb4fa02180032e4ad01b5f8b7192109bee67f7d3b27bbcb0fa150799cd835f"}
Oct 30 22:08:05.070766 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:05.070459    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr" event={"ID":"18b3281c-0a59-4598-b4cc-b3a4565b0c01","Type":"ContainerStarted","Data":"afbdc9dace32c428738c40f231e490a672099be48c1b492dbab2e1e2fe793fbe"}
Oct 30 22:08:05.185620 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:05.185564    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr" podStartSLOduration=2.185549242 podStartE2EDuration="2.185549242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:05.184394863 +0000 UTC m=+2079.658898840" watchObservedRunningTime="2026-10-30 22:08:05.185549242 +0000 UTC m=+2079.660053220"
Oct 30 22:08:05.340692 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:05.340664    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7051-be6eab"
Oct 30 22:08:05.488217 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:05.488184    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"]
Oct 30 22:08:05.488450 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:05.488424    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb" podUID="14b7cb87-8d56-4822-9d11-1cd8185c2732" containerName="istio-proxy" containerID="cri-o://cc78eda6b924355be42e10243a8354e8a89f0f2673094a96e1c6941913041f7f" gracePeriod=30
Oct 30 22:08:05.500767 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:05.500738    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd2c99d8-4f03-454a-9483-f6670a8875bd-kube-api-access-bnwsn\" (UniqueName: \"kubernetes.io/projected/bd2c99d8-4f03-454a-9483-f6670a8875bd-kube-api-access-bnwsn\") pod \"bd2c99d8-4f03-454a-9483-f6670a8875bd\" (UID: \"bd2c99d8-4f03-454a-9483-f6670a8875bd\") "
Oct 30 22:08:05.502934 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:05.502913    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2c99d8-4f03-454a-9483-f6670a8875bd-kube-api-access-bnwsn" pod "bd2c99d8-4f03-454a-9483-f6670a8875bd" (UID: "bd2c99d8-4f03-454a-9483-f6670a8875bd"). InnerVolumeSpecName "kube-api-access-bnwsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:05.603190 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:05.603103    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnwsn\" (UniqueName: \"kubernetes.io/projected/bd2c99d8-4f03-454a-9483-f6670a8875bd-kube-api-access-bnwsn\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:05.710534 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:05.710504    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd2c99d8-4f03-454a-9483-f6670a8875bd" path="/var/lib/kubelet/pods/bd2c99d8-4f03-454a-9483-f6670a8875bd/volumes"
Oct 30 22:08:06.078008 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:06.077976    5087 generic.go:356] "Generic (PLEG): container finished" podID="18b3281c-0a59-4598-b4cc-b3a4565b0c01" containerID="91fb4fa02180032e4ad01b5f8b7192109bee67f7d3b27bbcb0fa150799cd835f" exitCode=0
Oct 30 22:08:06.078500 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:06.078064    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr" event={"ID":"18b3281c-0a59-4598-b4cc-b3a4565b0c01","Type":"ContainerDied","Data":"91fb4fa02180032e4ad01b5f8b7192109bee67f7d3b27bbcb0fa150799cd835f"}
Oct 30 22:08:06.082884 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:06.082851    5087 generic.go:356] "Generic (PLEG): container finished" podID="bd2c99d8-4f03-454a-9483-f6670a8875bd" containerID="a1ca6bd9864b526ad0d6b13039294e87ddd791d5f375ce2ff0a9dc2519d7db5e" exitCode=137
Oct 30 22:08:06.083040 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:06.082915    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7051-be6eab"
Oct 30 22:08:06.083040 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:06.082921    5087 scope.go:122] "RemoveContainer" containerID="a1ca6bd9864b526ad0d6b13039294e87ddd791d5f375ce2ff0a9dc2519d7db5e"
Oct 30 22:08:06.094844 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:06.094823    5087 scope.go:122] "RemoveContainer" containerID="a1ca6bd9864b526ad0d6b13039294e87ddd791d5f375ce2ff0a9dc2519d7db5e"
Oct 30 22:08:06.095166 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:08:06.095138    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1ca6bd9864b526ad0d6b13039294e87ddd791d5f375ce2ff0a9dc2519d7db5e\": container with ID starting with a1ca6bd9864b526ad0d6b13039294e87ddd791d5f375ce2ff0a9dc2519d7db5e not found: ID does not exist" containerID="a1ca6bd9864b526ad0d6b13039294e87ddd791d5f375ce2ff0a9dc2519d7db5e"
Oct 30 22:08:06.095245 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:06.095181    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1ca6bd9864b526ad0d6b13039294e87ddd791d5f375ce2ff0a9dc2519d7db5e"} err="failed to get container status \"a1ca6bd9864b526ad0d6b13039294e87ddd791d5f375ce2ff0a9dc2519d7db5e\": rpc error: code = NotFound desc = could not find container \"a1ca6bd9864b526ad0d6b13039294e87ddd791d5f375ce2ff0a9dc2519d7db5e\": container with ID starting with a1ca6bd9864b526ad0d6b13039294e87ddd791d5f375ce2ff0a9dc2519d7db5e not found: ID does not exist"
Oct 30 22:08:07.258385 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:07.258337    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr"
Oct 30 22:08:07.427462 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:07.427371    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18b3281c-0a59-4598-b4cc-b3a4565b0c01-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18b3281c-0a59-4598-b4cc-b3a4565b0c01-maas-api-xks-ca\") pod \"18b3281c-0a59-4598-b4cc-b3a4565b0c01\" (UID: \"18b3281c-0a59-4598-b4cc-b3a4565b0c01\") "
Oct 30 22:08:07.427462 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:07.427438    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18b3281c-0a59-4598-b4cc-b3a4565b0c01-kube-api-access-nn8qg\" (UniqueName: \"kubernetes.io/projected/18b3281c-0a59-4598-b4cc-b3a4565b0c01-kube-api-access-nn8qg\") pod \"18b3281c-0a59-4598-b4cc-b3a4565b0c01\" (UID: \"18b3281c-0a59-4598-b4cc-b3a4565b0c01\") "
Oct 30 22:08:07.427758 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:07.427470    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18b3281c-0a59-4598-b4cc-b3a4565b0c01-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18b3281c-0a59-4598-b4cc-b3a4565b0c01-maas-api-openshift-ca\") pod \"18b3281c-0a59-4598-b4cc-b3a4565b0c01\" (UID: \"18b3281c-0a59-4598-b4cc-b3a4565b0c01\") "
Oct 30 22:08:07.427967 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:07.427934    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18b3281c-0a59-4598-b4cc-b3a4565b0c01-maas-api-openshift-ca" pod "18b3281c-0a59-4598-b4cc-b3a4565b0c01" (UID: "18b3281c-0a59-4598-b4cc-b3a4565b0c01"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:07.428258 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:07.428240    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18b3281c-0a59-4598-b4cc-b3a4565b0c01-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:07.429631 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:07.429605    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18b3281c-0a59-4598-b4cc-b3a4565b0c01-kube-api-access-nn8qg" pod "18b3281c-0a59-4598-b4cc-b3a4565b0c01" (UID: "18b3281c-0a59-4598-b4cc-b3a4565b0c01"). InnerVolumeSpecName "kube-api-access-nn8qg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:07.429862 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:07.429841    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18b3281c-0a59-4598-b4cc-b3a4565b0c01-maas-api-xks-ca" pod "18b3281c-0a59-4598-b4cc-b3a4565b0c01" (UID: "18b3281c-0a59-4598-b4cc-b3a4565b0c01"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:07.529381 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:07.529327    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18b3281c-0a59-4598-b4cc-b3a4565b0c01-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:07.529381 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:07.529377    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn8qg\" (UniqueName: \"kubernetes.io/projected/18b3281c-0a59-4598-b4cc-b3a4565b0c01-kube-api-access-nn8qg\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:08.101076 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:08.101045    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr"
Oct 30 22:08:08.101264 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:08.101045    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr" event={"ID":"18b3281c-0a59-4598-b4cc-b3a4565b0c01","Type":"ContainerDied","Data":"afbdc9dace32c428738c40f231e490a672099be48c1b492dbab2e1e2fe793fbe"}
Oct 30 22:08:08.101264 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:08.101173    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afbdc9dace32c428738c40f231e490a672099be48c1b492dbab2e1e2fe793fbe"
Oct 30 22:08:09.051921 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:09.051878    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj"]
Oct 30 22:08:09.052993 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:09.052969    5087 cpu_manager.go:414] "containerMap: removing container" podUID="18b3281c-0a59-4598-b4cc-b3a4565b0c01" containerName="revoke-keys"
Oct 30 22:08:09.052993 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:09.052992    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18b3281c-0a59-4598-b4cc-b3a4565b0c01" containerName="revoke-keys"
Oct 30 22:08:09.053178 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:09.053166    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="18b3281c-0a59-4598-b4cc-b3a4565b0c01" containerName="revoke-keys"
Oct 30 22:08:09.057384 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:09.057365    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj"
Oct 30 22:08:09.109026 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:09.108999    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-9c8c4d-serving-cert\""
Oct 30 22:08:09.151030 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:09.150998    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7372b3a2-49ea-42b8-9feb-48b67269aaad-metrics-certs\") pod \"maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj\" (UID: \"7372b3a2-49ea-42b8-9feb-48b67269aaad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj"
Oct 30 22:08:09.153013 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:09.152989    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fblzv\" (UniqueName: \"kubernetes.io/projected/7372b3a2-49ea-42b8-9feb-48b67269aaad-kube-api-access-fblzv\") pod \"maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj\" (UID: \"7372b3a2-49ea-42b8-9feb-48b67269aaad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj"
Oct 30 22:08:09.153211 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:09.153195    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7372b3a2-49ea-42b8-9feb-48b67269aaad-maas-api-tls\") pod \"maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj\" (UID: \"7372b3a2-49ea-42b8-9feb-48b67269aaad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj"
Oct 30 22:08:09.153509 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:09.153489    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7372b3a2-49ea-42b8-9feb-48b67269aaad-openshift-service-ca\") pod \"maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj\" (UID: \"7372b3a2-49ea-42b8-9feb-48b67269aaad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj"
Oct 30 22:08:09.164061 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:09.164039    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj"]
Oct 30 22:08:09.255128 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:09.255094    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7372b3a2-49ea-42b8-9feb-48b67269aaad-metrics-certs\") pod \"maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj\" (UID: \"7372b3a2-49ea-42b8-9feb-48b67269aaad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj"
Oct 30 22:08:09.255291 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:09.255245    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fblzv\" (UniqueName: \"kubernetes.io/projected/7372b3a2-49ea-42b8-9feb-48b67269aaad-kube-api-access-fblzv\") pod \"maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj\" (UID: \"7372b3a2-49ea-42b8-9feb-48b67269aaad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj"
Oct 30 22:08:09.255291 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:09.255276    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7372b3a2-49ea-42b8-9feb-48b67269aaad-maas-api-tls\") pod \"maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj\" (UID: \"7372b3a2-49ea-42b8-9feb-48b67269aaad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj"
Oct 30 22:08:09.255481 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:08:09.255458    5087 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-serving-cert: secret "maas-api-e2e-api-a-9c8c4d-serving-cert" not found
Oct 30 22:08:09.255536 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:09.255490    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7372b3a2-49ea-42b8-9feb-48b67269aaad-openshift-service-ca\") pod \"maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj\" (UID: \"7372b3a2-49ea-42b8-9feb-48b67269aaad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj"
Oct 30 22:08:09.255600 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:08:09.255543    5087 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7372b3a2-49ea-42b8-9feb-48b67269aaad-maas-api-tls podName:7372b3a2-49ea-42b8-9feb-48b67269aaad nodeName:}" failed. No retries permitted until 2026-10-30 22:08:09.755523525 +0000 UTC m=+2084.230027500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7372b3a2-49ea-42b8-9feb-48b67269aaad-maas-api-tls") pod "maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj" (UID: "7372b3a2-49ea-42b8-9feb-48b67269aaad") : secret "maas-api-e2e-api-a-9c8c4d-serving-cert" not found
Oct 30 22:08:09.256543 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:09.256505    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7372b3a2-49ea-42b8-9feb-48b67269aaad-openshift-service-ca\") pod \"maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj\" (UID: \"7372b3a2-49ea-42b8-9feb-48b67269aaad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj"
Oct 30 22:08:09.257963 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:09.257937    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7372b3a2-49ea-42b8-9feb-48b67269aaad-metrics-certs\") pod \"maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj\" (UID: \"7372b3a2-49ea-42b8-9feb-48b67269aaad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj"
Oct 30 22:08:09.314744 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:09.314662    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fblzv\" (UniqueName: \"kubernetes.io/projected/7372b3a2-49ea-42b8-9feb-48b67269aaad-kube-api-access-fblzv\") pod \"maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj\" (UID: \"7372b3a2-49ea-42b8-9feb-48b67269aaad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj"
Oct 30 22:08:09.762715 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:09.762681    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7372b3a2-49ea-42b8-9feb-48b67269aaad-maas-api-tls\") pod \"maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj\" (UID: \"7372b3a2-49ea-42b8-9feb-48b67269aaad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj"
Oct 30 22:08:09.765336 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:09.765313    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7372b3a2-49ea-42b8-9feb-48b67269aaad-maas-api-tls\") pod \"maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj\" (UID: \"7372b3a2-49ea-42b8-9feb-48b67269aaad\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj"
Oct 30 22:08:09.973191 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:09.973156    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj"
Oct 30 22:08:10.352831 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:10.352792    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj"]
Oct 30 22:08:10.353625 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:08:10.353597    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7372b3a2_49ea_42b8_9feb_48b67269aaad.slice/crio-86ffbd240b2f3922253540edc22119820c3b16cfdd9ef87f5449d28b4786e001 WatchSource:0}: Error finding container 86ffbd240b2f3922253540edc22119820c3b16cfdd9ef87f5449d28b4786e001: Status 404 returned error can't find the container with id 86ffbd240b2f3922253540edc22119820c3b16cfdd9ef87f5449d28b4786e001
Oct 30 22:08:10.796685 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:10.796663    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 22:08:10.978804 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:10.978705    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14b7cb87-8d56-4822-9d11-1cd8185c2732-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b7cb87-8d56-4822-9d11-1cd8185c2732-istiod-ca-cert\") pod \"14b7cb87-8d56-4822-9d11-1cd8185c2732\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") "
Oct 30 22:08:10.978976 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:10.978842    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14b7cb87-8d56-4822-9d11-1cd8185c2732-kube-api-access-lhcjt\" (UniqueName: \"kubernetes.io/projected/14b7cb87-8d56-4822-9d11-1cd8185c2732-kube-api-access-lhcjt\") pod \"14b7cb87-8d56-4822-9d11-1cd8185c2732\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") "
Oct 30 22:08:10.978976 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:10.978879    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-workload-certs\") pod \"14b7cb87-8d56-4822-9d11-1cd8185c2732\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") "
Oct 30 22:08:10.979074 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:10.979003    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14b7cb87-8d56-4822-9d11-1cd8185c2732-istio-token\" (UniqueName: \"kubernetes.io/projected/14b7cb87-8d56-4822-9d11-1cd8185c2732-istio-token\") pod \"14b7cb87-8d56-4822-9d11-1cd8185c2732\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") "
Oct 30 22:08:10.979074 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:10.979067    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-workload-socket\") pod \"14b7cb87-8d56-4822-9d11-1cd8185c2732\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") "
Oct 30 22:08:10.979136 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:10.979114    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14b7cb87-8d56-4822-9d11-1cd8185c2732-istiod-ca-cert" pod "14b7cb87-8d56-4822-9d11-1cd8185c2732" (UID: "14b7cb87-8d56-4822-9d11-1cd8185c2732"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:10.979171 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:10.979135    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-istio-data\") pod \"14b7cb87-8d56-4822-9d11-1cd8185c2732\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") "
Oct 30 22:08:10.979223 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:10.979172    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-istio-envoy\") pod \"14b7cb87-8d56-4822-9d11-1cd8185c2732\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") "
Oct 30 22:08:10.979282 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:10.979266    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-credential-socket\") pod \"14b7cb87-8d56-4822-9d11-1cd8185c2732\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") "
Oct 30 22:08:10.979365 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:10.979307    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14b7cb87-8d56-4822-9d11-1cd8185c2732-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b7cb87-8d56-4822-9d11-1cd8185c2732-istio-podinfo\") pod \"14b7cb87-8d56-4822-9d11-1cd8185c2732\" (UID: \"14b7cb87-8d56-4822-9d11-1cd8185c2732\") "
Oct 30 22:08:10.979365 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:10.979311    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-workload-certs" pod "14b7cb87-8d56-4822-9d11-1cd8185c2732" (UID: "14b7cb87-8d56-4822-9d11-1cd8185c2732"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:10.979842 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:10.979818    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-istio-data" pod "14b7cb87-8d56-4822-9d11-1cd8185c2732" (UID: "14b7cb87-8d56-4822-9d11-1cd8185c2732"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:10.980017 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:10.979869    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b7cb87-8d56-4822-9d11-1cd8185c2732-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:10.980101 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:10.980033    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:10.980266 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:10.980247    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-workload-socket" pod "14b7cb87-8d56-4822-9d11-1cd8185c2732" (UID: "14b7cb87-8d56-4822-9d11-1cd8185c2732"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:10.980548 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:10.980522    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-credential-socket" pod "14b7cb87-8d56-4822-9d11-1cd8185c2732" (UID: "14b7cb87-8d56-4822-9d11-1cd8185c2732"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:10.984857 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:10.984824    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-istio-envoy" pod "14b7cb87-8d56-4822-9d11-1cd8185c2732" (UID: "14b7cb87-8d56-4822-9d11-1cd8185c2732"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:10.984961 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:10.984907    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b7cb87-8d56-4822-9d11-1cd8185c2732-kube-api-access-lhcjt" pod "14b7cb87-8d56-4822-9d11-1cd8185c2732" (UID: "14b7cb87-8d56-4822-9d11-1cd8185c2732"). InnerVolumeSpecName "kube-api-access-lhcjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:10.984961 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:10.984928    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14b7cb87-8d56-4822-9d11-1cd8185c2732-istio-podinfo" pod "14b7cb87-8d56-4822-9d11-1cd8185c2732" (UID: "14b7cb87-8d56-4822-9d11-1cd8185c2732"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:08:10.985228 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:10.985209    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b7cb87-8d56-4822-9d11-1cd8185c2732-istio-token" pod "14b7cb87-8d56-4822-9d11-1cd8185c2732" (UID: "14b7cb87-8d56-4822-9d11-1cd8185c2732"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:11.080984 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:11.080938    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14b7cb87-8d56-4822-9d11-1cd8185c2732-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:11.080984 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:11.080971    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:11.080984 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:11.080981    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:11.080984 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:11.080992    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:11.081371 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:11.081001    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b7cb87-8d56-4822-9d11-1cd8185c2732-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:11.081371 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:11.081010    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b7cb87-8d56-4822-9d11-1cd8185c2732-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:11.081371 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:11.081018    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhcjt\" (UniqueName: \"kubernetes.io/projected/14b7cb87-8d56-4822-9d11-1cd8185c2732-kube-api-access-lhcjt\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:11.122603 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:11.122569    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj" event={"ID":"7372b3a2-49ea-42b8-9feb-48b67269aaad","Type":"ContainerStarted","Data":"86ffbd240b2f3922253540edc22119820c3b16cfdd9ef87f5449d28b4786e001"}
Oct 30 22:08:11.124121 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:11.124091    5087 generic.go:356] "Generic (PLEG): container finished" podID="14b7cb87-8d56-4822-9d11-1cd8185c2732" containerID="cc78eda6b924355be42e10243a8354e8a89f0f2673094a96e1c6941913041f7f" exitCode=0
Oct 30 22:08:11.124269 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:11.124151    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"
Oct 30 22:08:11.124269 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:11.124182    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb" event={"ID":"14b7cb87-8d56-4822-9d11-1cd8185c2732","Type":"ContainerDied","Data":"cc78eda6b924355be42e10243a8354e8a89f0f2673094a96e1c6941913041f7f"}
Oct 30 22:08:11.124269 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:11.124224    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb" event={"ID":"14b7cb87-8d56-4822-9d11-1cd8185c2732","Type":"ContainerDied","Data":"1335464d56cdf8d67d77930d1ad40f0e0413036decd4e48b5ec072a500d2e237"}
Oct 30 22:08:11.124269 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:11.124253    5087 scope.go:122] "RemoveContainer" containerID="cc78eda6b924355be42e10243a8354e8a89f0f2673094a96e1c6941913041f7f"
Oct 30 22:08:11.148010 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:11.147985    5087 scope.go:122] "RemoveContainer" containerID="cc78eda6b924355be42e10243a8354e8a89f0f2673094a96e1c6941913041f7f"
Oct 30 22:08:11.148421 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:08:11.148394    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc78eda6b924355be42e10243a8354e8a89f0f2673094a96e1c6941913041f7f\": container with ID starting with cc78eda6b924355be42e10243a8354e8a89f0f2673094a96e1c6941913041f7f not found: ID does not exist" containerID="cc78eda6b924355be42e10243a8354e8a89f0f2673094a96e1c6941913041f7f"
Oct 30 22:08:11.148523 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:11.148432    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc78eda6b924355be42e10243a8354e8a89f0f2673094a96e1c6941913041f7f"} err="failed to get container status \"cc78eda6b924355be42e10243a8354e8a89f0f2673094a96e1c6941913041f7f\": rpc error: code = NotFound desc = could not find container \"cc78eda6b924355be42e10243a8354e8a89f0f2673094a96e1c6941913041f7f\": container with ID starting with cc78eda6b924355be42e10243a8354e8a89f0f2673094a96e1c6941913041f7f not found: ID does not exist"
Oct 30 22:08:11.299957 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:11.299909    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"]
Oct 30 22:08:11.303919 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:11.303895    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-xzvzb"]
Oct 30 22:08:11.710686 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:11.710604    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14b7cb87-8d56-4822-9d11-1cd8185c2732" path="/var/lib/kubelet/pods/14b7cb87-8d56-4822-9d11-1cd8185c2732/volumes"
Oct 30 22:08:12.146618 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:12.146580    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj" event={"ID":"7372b3a2-49ea-42b8-9feb-48b67269aaad","Type":"ContainerStarted","Data":"9d2b96e1dd3ff69d47019425297b7fb53d8bc77e4b765bb014b8702435d903bd"}
Oct 30 22:08:12.146876 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:12.146846    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj"
Oct 30 22:08:12.262022 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:12.261953    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj" podStartSLOduration=2.465118467 podStartE2EDuration="3.261934139s" totalImagesPullingTime="796.815672ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:12.261148029 +0000 UTC m=+2086.735652021" watchObservedRunningTime="2026-10-30 22:08:12.261934139 +0000 UTC m=+2086.736438118"
Oct 30 22:08:17.169291 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:08:17.169249    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:08:18.161157 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:18.161129    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj"
Oct 30 22:08:26.053685 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:26.053643    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 22:08:26.066776 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:26.066547    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 22:08:26.070413 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:26.070390    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 22:08:26.080897 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:26.080874    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 22:08:28.903942 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:28.903911    5087 scope.go:122] "RemoveContainer" containerID="eb9ffab2bd6d62f75f0a19f40a8207d86d078c05e2437dc0544cc4e633c17ce3"
Oct 30 22:08:28.913934 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:28.913915    5087 scope.go:122] "RemoveContainer" containerID="3429564b1fb68fe989502687cec26043a60068a7c9e1696376e32a1bacbfe8c2"
Oct 30 22:08:32.618875 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:32.618836    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"]
Oct 30 22:08:32.619294 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:32.619045    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt" podUID="625ae489-24f8-4985-a6d8-6b3c2953a8af" containerName="discovery" containerID="cri-o://10ed95496956b5d64e77e7d3864c4e9c379b597a2ee28de0b0aea0709bb20629" gracePeriod=30
Oct 30 22:08:32.978998 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:32.978969    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"
Oct 30 22:08:33.084309 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.084272    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/625ae489-24f8-4985-a6d8-6b3c2953a8af-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/625ae489-24f8-4985-a6d8-6b3c2953a8af-istio-kubeconfig\") pod \"625ae489-24f8-4985-a6d8-6b3c2953a8af\" (UID: \"625ae489-24f8-4985-a6d8-6b3c2953a8af\") "
Oct 30 22:08:33.084503 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.084339    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/625ae489-24f8-4985-a6d8-6b3c2953a8af-kube-api-access-7jtj9\" (UniqueName: \"kubernetes.io/projected/625ae489-24f8-4985-a6d8-6b3c2953a8af-kube-api-access-7jtj9\") pod \"625ae489-24f8-4985-a6d8-6b3c2953a8af\" (UID: \"625ae489-24f8-4985-a6d8-6b3c2953a8af\") "
Oct 30 22:08:33.084503 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.084425    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/625ae489-24f8-4985-a6d8-6b3c2953a8af-istio-token\" (UniqueName: \"kubernetes.io/projected/625ae489-24f8-4985-a6d8-6b3c2953a8af-istio-token\") pod \"625ae489-24f8-4985-a6d8-6b3c2953a8af\" (UID: \"625ae489-24f8-4985-a6d8-6b3c2953a8af\") "
Oct 30 22:08:33.084503 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.084452    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/625ae489-24f8-4985-a6d8-6b3c2953a8af-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/625ae489-24f8-4985-a6d8-6b3c2953a8af-istio-csr-dns-cert\") pod \"625ae489-24f8-4985-a6d8-6b3c2953a8af\" (UID: \"625ae489-24f8-4985-a6d8-6b3c2953a8af\") "
Oct 30 22:08:33.084503 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.084477    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/625ae489-24f8-4985-a6d8-6b3c2953a8af-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/625ae489-24f8-4985-a6d8-6b3c2953a8af-istio-csr-ca-configmap\") pod \"625ae489-24f8-4985-a6d8-6b3c2953a8af\" (UID: \"625ae489-24f8-4985-a6d8-6b3c2953a8af\") "
Oct 30 22:08:33.084730 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.084635    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/625ae489-24f8-4985-a6d8-6b3c2953a8af-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/625ae489-24f8-4985-a6d8-6b3c2953a8af-local-certs\") pod \"625ae489-24f8-4985-a6d8-6b3c2953a8af\" (UID: \"625ae489-24f8-4985-a6d8-6b3c2953a8af\") "
Oct 30 22:08:33.084730 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.084691    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/625ae489-24f8-4985-a6d8-6b3c2953a8af-cacerts\" (UniqueName: \"kubernetes.io/secret/625ae489-24f8-4985-a6d8-6b3c2953a8af-cacerts\") pod \"625ae489-24f8-4985-a6d8-6b3c2953a8af\" (UID: \"625ae489-24f8-4985-a6d8-6b3c2953a8af\") "
Oct 30 22:08:33.084913 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.084873    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/625ae489-24f8-4985-a6d8-6b3c2953a8af-istio-csr-ca-configmap" pod "625ae489-24f8-4985-a6d8-6b3c2953a8af" (UID: "625ae489-24f8-4985-a6d8-6b3c2953a8af"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:33.085593 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.085557    5087 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/625ae489-24f8-4985-a6d8-6b3c2953a8af-istio-csr-ca-configmap\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:33.087408 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.087374    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625ae489-24f8-4985-a6d8-6b3c2953a8af-kube-api-access-7jtj9" pod "625ae489-24f8-4985-a6d8-6b3c2953a8af" (UID: "625ae489-24f8-4985-a6d8-6b3c2953a8af"). InnerVolumeSpecName "kube-api-access-7jtj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:33.087981 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.087875    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/625ae489-24f8-4985-a6d8-6b3c2953a8af-istio-kubeconfig" pod "625ae489-24f8-4985-a6d8-6b3c2953a8af" (UID: "625ae489-24f8-4985-a6d8-6b3c2953a8af"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:33.087981 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.087917    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/625ae489-24f8-4985-a6d8-6b3c2953a8af-istio-csr-dns-cert" pod "625ae489-24f8-4985-a6d8-6b3c2953a8af" (UID: "625ae489-24f8-4985-a6d8-6b3c2953a8af"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:33.087981 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.087945    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/625ae489-24f8-4985-a6d8-6b3c2953a8af-cacerts" pod "625ae489-24f8-4985-a6d8-6b3c2953a8af" (UID: "625ae489-24f8-4985-a6d8-6b3c2953a8af"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:33.087981 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.087956    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625ae489-24f8-4985-a6d8-6b3c2953a8af-istio-token" pod "625ae489-24f8-4985-a6d8-6b3c2953a8af" (UID: "625ae489-24f8-4985-a6d8-6b3c2953a8af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:33.088202 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.088017    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/625ae489-24f8-4985-a6d8-6b3c2953a8af-local-certs" pod "625ae489-24f8-4985-a6d8-6b3c2953a8af" (UID: "625ae489-24f8-4985-a6d8-6b3c2953a8af"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:33.187044 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.186964    5087 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/625ae489-24f8-4985-a6d8-6b3c2953a8af-cacerts\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:33.187044 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.186994    5087 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/625ae489-24f8-4985-a6d8-6b3c2953a8af-istio-kubeconfig\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:33.187044 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.187004    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jtj9\" (UniqueName: \"kubernetes.io/projected/625ae489-24f8-4985-a6d8-6b3c2953a8af-kube-api-access-7jtj9\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:33.187044 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.187014    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/625ae489-24f8-4985-a6d8-6b3c2953a8af-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:33.187044 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.187022    5087 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/625ae489-24f8-4985-a6d8-6b3c2953a8af-istio-csr-dns-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:33.187044 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.187030    5087 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/625ae489-24f8-4985-a6d8-6b3c2953a8af-local-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:33.294114 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.294078    5087 generic.go:356] "Generic (PLEG): container finished" podID="625ae489-24f8-4985-a6d8-6b3c2953a8af" containerID="10ed95496956b5d64e77e7d3864c4e9c379b597a2ee28de0b0aea0709bb20629" exitCode=0
Oct 30 22:08:33.294267 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.294129    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"
Oct 30 22:08:33.294267 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.294167    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt" event={"ID":"625ae489-24f8-4985-a6d8-6b3c2953a8af","Type":"ContainerDied","Data":"10ed95496956b5d64e77e7d3864c4e9c379b597a2ee28de0b0aea0709bb20629"}
Oct 30 22:08:33.294267 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.294220    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt" event={"ID":"625ae489-24f8-4985-a6d8-6b3c2953a8af","Type":"ContainerDied","Data":"a79a404144167520fc6844c3cfe10a336e2d520da6f2ad32508fbfafde7f72df"}
Oct 30 22:08:33.294267 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.294243    5087 scope.go:122] "RemoveContainer" containerID="10ed95496956b5d64e77e7d3864c4e9c379b597a2ee28de0b0aea0709bb20629"
Oct 30 22:08:33.305858 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.305841    5087 scope.go:122] "RemoveContainer" containerID="10ed95496956b5d64e77e7d3864c4e9c379b597a2ee28de0b0aea0709bb20629"
Oct 30 22:08:33.306117 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:08:33.306096    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10ed95496956b5d64e77e7d3864c4e9c379b597a2ee28de0b0aea0709bb20629\": container with ID starting with 10ed95496956b5d64e77e7d3864c4e9c379b597a2ee28de0b0aea0709bb20629 not found: ID does not exist" containerID="10ed95496956b5d64e77e7d3864c4e9c379b597a2ee28de0b0aea0709bb20629"
Oct 30 22:08:33.306156 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.306127    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10ed95496956b5d64e77e7d3864c4e9c379b597a2ee28de0b0aea0709bb20629"} err="failed to get container status \"10ed95496956b5d64e77e7d3864c4e9c379b597a2ee28de0b0aea0709bb20629\": rpc error: code = NotFound desc = could not find container \"10ed95496956b5d64e77e7d3864c4e9c379b597a2ee28de0b0aea0709bb20629\": container with ID starting with 10ed95496956b5d64e77e7d3864c4e9c379b597a2ee28de0b0aea0709bb20629 not found: ID does not exist"
Oct 30 22:08:33.477809 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.477727    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"]
Oct 30 22:08:33.493532 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.493498    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zctbt"]
Oct 30 22:08:33.710216 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:33.710185    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="625ae489-24f8-4985-a6d8-6b3c2953a8af" path="/var/lib/kubelet/pods/625ae489-24f8-4985-a6d8-6b3c2953a8af/volumes"
Oct 30 22:08:37.906876 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:37.906846    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb"]
Oct 30 22:08:37.907501 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:37.907485    5087 cpu_manager.go:414] "containerMap: removing container" podUID="625ae489-24f8-4985-a6d8-6b3c2953a8af" containerName="discovery"
Oct 30 22:08:37.907562 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:37.907502    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="625ae489-24f8-4985-a6d8-6b3c2953a8af" containerName="discovery"
Oct 30 22:08:37.907562 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:37.907523    5087 cpu_manager.go:414] "containerMap: removing container" podUID="14b7cb87-8d56-4822-9d11-1cd8185c2732" containerName="istio-proxy"
Oct 30 22:08:37.907562 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:37.907532    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14b7cb87-8d56-4822-9d11-1cd8185c2732" containerName="istio-proxy"
Oct 30 22:08:37.907738 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:37.907724    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="625ae489-24f8-4985-a6d8-6b3c2953a8af" containerName="discovery"
Oct 30 22:08:37.907772 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:37.907744    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="14b7cb87-8d56-4822-9d11-1cd8185c2732" containerName="istio-proxy"
Oct 30 22:08:37.912660 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:37.912642    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb"
Oct 30 22:08:38.017152 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:38.017117    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-78cf83-serving-cert\""
Oct 30 22:08:38.021752 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:38.021722    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb"]
Oct 30 22:08:38.072370 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:38.071377    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfthx\" (UniqueName: \"kubernetes.io/projected/6b3238c7-a167-49c6-8d68-6c7af7145853-kube-api-access-zfthx\") pod \"maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb\" (UID: \"6b3238c7-a167-49c6-8d68-6c7af7145853\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb"
Oct 30 22:08:38.072370 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:38.071503    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3238c7-a167-49c6-8d68-6c7af7145853-openshift-service-ca\") pod \"maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb\" (UID: \"6b3238c7-a167-49c6-8d68-6c7af7145853\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb"
Oct 30 22:08:38.072370 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:38.071602    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b3238c7-a167-49c6-8d68-6c7af7145853-metrics-certs\") pod \"maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb\" (UID: \"6b3238c7-a167-49c6-8d68-6c7af7145853\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb"
Oct 30 22:08:38.072370 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:38.071979    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b3238c7-a167-49c6-8d68-6c7af7145853-maas-api-tls\") pod \"maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb\" (UID: \"6b3238c7-a167-49c6-8d68-6c7af7145853\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb"
Oct 30 22:08:38.173960 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:38.173879    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b3238c7-a167-49c6-8d68-6c7af7145853-maas-api-tls\") pod \"maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb\" (UID: \"6b3238c7-a167-49c6-8d68-6c7af7145853\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb"
Oct 30 22:08:38.174139 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:38.174003    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfthx\" (UniqueName: \"kubernetes.io/projected/6b3238c7-a167-49c6-8d68-6c7af7145853-kube-api-access-zfthx\") pod \"maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb\" (UID: \"6b3238c7-a167-49c6-8d68-6c7af7145853\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb"
Oct 30 22:08:38.174139 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:38.174103    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3238c7-a167-49c6-8d68-6c7af7145853-openshift-service-ca\") pod \"maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb\" (UID: \"6b3238c7-a167-49c6-8d68-6c7af7145853\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb"
Oct 30 22:08:38.174339 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:38.174190    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b3238c7-a167-49c6-8d68-6c7af7145853-metrics-certs\") pod \"maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb\" (UID: \"6b3238c7-a167-49c6-8d68-6c7af7145853\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb"
Oct 30 22:08:38.175185 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:38.175155    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3238c7-a167-49c6-8d68-6c7af7145853-openshift-service-ca\") pod \"maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb\" (UID: \"6b3238c7-a167-49c6-8d68-6c7af7145853\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb"
Oct 30 22:08:38.177093 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:38.177069    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b3238c7-a167-49c6-8d68-6c7af7145853-maas-api-tls\") pod \"maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb\" (UID: \"6b3238c7-a167-49c6-8d68-6c7af7145853\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb"
Oct 30 22:08:38.177477 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:38.177452    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b3238c7-a167-49c6-8d68-6c7af7145853-metrics-certs\") pod \"maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb\" (UID: \"6b3238c7-a167-49c6-8d68-6c7af7145853\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb"
Oct 30 22:08:38.231333 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:38.231295    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfthx\" (UniqueName: \"kubernetes.io/projected/6b3238c7-a167-49c6-8d68-6c7af7145853-kube-api-access-zfthx\") pod \"maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb\" (UID: \"6b3238c7-a167-49c6-8d68-6c7af7145853\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb"
Oct 30 22:08:38.527850 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:38.527766    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb"
Oct 30 22:08:38.931316 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:38.931286    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb"]
Oct 30 22:08:38.932894 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:08:38.932866    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3238c7_a167_49c6_8d68_6c7af7145853.slice/crio-519e5ea31203c8f36db37dfbdf9a06ef8f5300a11344723ed3e785086e5b71e1 WatchSource:0}: Error finding container 519e5ea31203c8f36db37dfbdf9a06ef8f5300a11344723ed3e785086e5b71e1: Status 404 returned error can't find the container with id 519e5ea31203c8f36db37dfbdf9a06ef8f5300a11344723ed3e785086e5b71e1
Oct 30 22:08:39.337250 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:39.337203    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb" event={"ID":"6b3238c7-a167-49c6-8d68-6c7af7145853","Type":"ContainerStarted","Data":"519e5ea31203c8f36db37dfbdf9a06ef8f5300a11344723ed3e785086e5b71e1"}
Oct 30 22:08:40.348815 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:40.348780    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb" event={"ID":"6b3238c7-a167-49c6-8d68-6c7af7145853","Type":"ContainerStarted","Data":"b5187f68ca27e0bc6958daaef60d78261548a1bfd19772c5833cb5b454bd443c"}
Oct 30 22:08:40.349316 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:40.349012    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb"
Oct 30 22:08:40.469560 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:40.469489    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb" podStartSLOduration=2.693260559 podStartE2EDuration="3.469470473s" totalImagesPullingTime="776.209914ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:40.465298597 +0000 UTC m=+2114.939802577" watchObservedRunningTime="2026-10-30 22:08:40.469470473 +0000 UTC m=+2114.943974453"
Oct 30 22:08:41.044562 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:41.044535    5087 patch_prober.go:29] interesting pod/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": read tcp 10.132.0.2:36490->10.132.0.159:15021: read: connection reset by peer" start-of-body=
Oct 30 22:08:41.044704 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:41.044595    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82" podUID="c975ae55-4f25-494b-acd0-7458374253df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": read tcp 10.132.0.2:36490->10.132.0.159:15021: read: connection reset by peer"
Oct 30 22:08:41.359745 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:41.359659    5087 generic.go:356] "Generic (PLEG): container finished" podID="c975ae55-4f25-494b-acd0-7458374253df" containerID="ed425bb89ff342c4a1061e01a0f7e2011ab1194812c0b379a84bdba67dd54487" exitCode=0
Oct 30 22:08:41.359745 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:41.359731    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82" event={"ID":"c975ae55-4f25-494b-acd0-7458374253df","Type":"ContainerDied","Data":"ed425bb89ff342c4a1061e01a0f7e2011ab1194812c0b379a84bdba67dd54487"}
Oct 30 22:08:41.360274 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:41.360200    5087 scope.go:122] "RemoveContainer" containerID="ed425bb89ff342c4a1061e01a0f7e2011ab1194812c0b379a84bdba67dd54487"
Oct 30 22:08:41.362204 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:41.362170    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:08:41.362292 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:41.362266    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:08:41.362334 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:41.362315    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:08:42.381517 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:42.381480    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82" event={"ID":"c975ae55-4f25-494b-acd0-7458374253df","Type":"ContainerStarted","Data":"cf0513a665aab5adf830e5842a34964c8a2609604c7220ea411abf6df9199c55"}
Oct 30 22:08:42.900452 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:42.900413    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:08:42.900731 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:42.900708    5087 patch_prober.go:29] interesting pod/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 22:08:42.900797 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:42.900755    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82" podUID="c975ae55-4f25-494b-acd0-7458374253df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 22:08:43.900681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:43.900642    5087 patch_prober.go:29] interesting pod/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 22:08:43.901071 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:43.900697    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82" podUID="c975ae55-4f25-494b-acd0-7458374253df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 22:08:44.900334 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:44.900294    5087 patch_prober.go:29] interesting pod/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 22:08:44.900538 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:44.900397    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82" podUID="c975ae55-4f25-494b-acd0-7458374253df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 22:08:45.900057 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:45.900018    5087 patch_prober.go:29] interesting pod/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 22:08:45.900505 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:45.900078    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82" podUID="c975ae55-4f25-494b-acd0-7458374253df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 22:08:46.373473 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:46.373429    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb"
Oct 30 22:08:46.900894 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:46.900854    5087 patch_prober.go:29] interesting pod/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body=
Oct 30 22:08:46.901301 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:46.900917    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82" podUID="c975ae55-4f25-494b-acd0-7458374253df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused"
Oct 30 22:08:47.194216 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:08:47.194108    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:08:47.904964 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:47.904934    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:08:47.905435 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:47.905416    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:08:47.906211 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:47.906193    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:08:50.790889 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:50.790857    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9d4043-927m5"]
Oct 30 22:08:50.794803 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:50.794786    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9d4043-927m5"
Oct 30 22:08:50.846386 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:50.846326    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:08:50.880385 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:50.880334    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ba89675-526d-4f5d-b8ff-d676460179fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9d4043-927m5\" (UID: \"3ba89675-526d-4f5d-b8ff-d676460179fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9d4043-927m5"
Oct 30 22:08:50.880574 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:50.880434    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8frqq\" (UniqueName: \"kubernetes.io/projected/3ba89675-526d-4f5d-b8ff-d676460179fd-kube-api-access-8frqq\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9d4043-927m5\" (UID: \"3ba89675-526d-4f5d-b8ff-d676460179fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9d4043-927m5"
Oct 30 22:08:50.880574 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:50.880550    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ba89675-526d-4f5d-b8ff-d676460179fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9d4043-927m5\" (UID: \"3ba89675-526d-4f5d-b8ff-d676460179fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9d4043-927m5"
Oct 30 22:08:50.896515 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:50.896487    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:08:50.899714 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:50.899685    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9d4043-927m5"]
Oct 30 22:08:50.981479 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:50.981440    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ba89675-526d-4f5d-b8ff-d676460179fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9d4043-927m5\" (UID: \"3ba89675-526d-4f5d-b8ff-d676460179fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9d4043-927m5"
Oct 30 22:08:50.981690 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:50.981659    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ba89675-526d-4f5d-b8ff-d676460179fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9d4043-927m5\" (UID: \"3ba89675-526d-4f5d-b8ff-d676460179fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9d4043-927m5"
Oct 30 22:08:50.981772 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:50.981717    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8frqq\" (UniqueName: \"kubernetes.io/projected/3ba89675-526d-4f5d-b8ff-d676460179fd-kube-api-access-8frqq\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9d4043-927m5\" (UID: \"3ba89675-526d-4f5d-b8ff-d676460179fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9d4043-927m5"
Oct 30 22:08:50.982218 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:50.982193    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ba89675-526d-4f5d-b8ff-d676460179fd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9d4043-927m5\" (UID: \"3ba89675-526d-4f5d-b8ff-d676460179fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9d4043-927m5"
Oct 30 22:08:50.984282 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:50.984263    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ba89675-526d-4f5d-b8ff-d676460179fd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9d4043-927m5\" (UID: \"3ba89675-526d-4f5d-b8ff-d676460179fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9d4043-927m5"
Oct 30 22:08:51.039653 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:51.039622    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8frqq\" (UniqueName: \"kubernetes.io/projected/3ba89675-526d-4f5d-b8ff-d676460179fd-kube-api-access-8frqq\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-9d4043-927m5\" (UID: \"3ba89675-526d-4f5d-b8ff-d676460179fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9d4043-927m5"
Oct 30 22:08:51.108784 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:51.108718    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9d4043-927m5"
Oct 30 22:08:51.484622 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:51.484586    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9d4043-927m5"]
Oct 30 22:08:51.486718 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:08:51.486680    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba89675_526d_4f5d_b8ff_d676460179fd.slice/crio-6434683688a26389b34f6a92e61f7cca293eb97ad0bb4f6bdfe422b446a2ad25 WatchSource:0}: Error finding container 6434683688a26389b34f6a92e61f7cca293eb97ad0bb4f6bdfe422b446a2ad25: Status 404 returned error can't find the container with id 6434683688a26389b34f6a92e61f7cca293eb97ad0bb4f6bdfe422b446a2ad25
Oct 30 22:08:52.451595 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:52.451564    5087 generic.go:356] "Generic (PLEG): container finished" podID="3ba89675-526d-4f5d-b8ff-d676460179fd" containerID="f890fdea7e844b91994f604ebe7fc3a8714cf38278576f8d44893f2d3210c542" exitCode=0
Oct 30 22:08:52.452006 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:52.451647    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9d4043-927m5" event={"ID":"3ba89675-526d-4f5d-b8ff-d676460179fd","Type":"ContainerDied","Data":"f890fdea7e844b91994f604ebe7fc3a8714cf38278576f8d44893f2d3210c542"}
Oct 30 22:08:52.452006 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:52.451704    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9d4043-927m5" event={"ID":"3ba89675-526d-4f5d-b8ff-d676460179fd","Type":"ContainerStarted","Data":"6434683688a26389b34f6a92e61f7cca293eb97ad0bb4f6bdfe422b446a2ad25"}
Oct 30 22:08:53.623222 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:53.623195    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9d4043-927m5"
Oct 30 22:08:53.714863 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:53.714795    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ba89675-526d-4f5d-b8ff-d676460179fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ba89675-526d-4f5d-b8ff-d676460179fd-maas-api-openshift-ca\") pod \"3ba89675-526d-4f5d-b8ff-d676460179fd\" (UID: \"3ba89675-526d-4f5d-b8ff-d676460179fd\") "
Oct 30 22:08:53.714863 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:53.714851    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ba89675-526d-4f5d-b8ff-d676460179fd-kube-api-access-8frqq\" (UniqueName: \"kubernetes.io/projected/3ba89675-526d-4f5d-b8ff-d676460179fd-kube-api-access-8frqq\") pod \"3ba89675-526d-4f5d-b8ff-d676460179fd\" (UID: \"3ba89675-526d-4f5d-b8ff-d676460179fd\") "
Oct 30 22:08:53.715073 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:53.714966    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ba89675-526d-4f5d-b8ff-d676460179fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ba89675-526d-4f5d-b8ff-d676460179fd-maas-api-xks-ca\") pod \"3ba89675-526d-4f5d-b8ff-d676460179fd\" (UID: \"3ba89675-526d-4f5d-b8ff-d676460179fd\") "
Oct 30 22:08:53.715197 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:53.715171    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ba89675-526d-4f5d-b8ff-d676460179fd-maas-api-openshift-ca" pod "3ba89675-526d-4f5d-b8ff-d676460179fd" (UID: "3ba89675-526d-4f5d-b8ff-d676460179fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:53.715486 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:53.715468    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ba89675-526d-4f5d-b8ff-d676460179fd-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:53.717120 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:53.717092    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ba89675-526d-4f5d-b8ff-d676460179fd-maas-api-xks-ca" pod "3ba89675-526d-4f5d-b8ff-d676460179fd" (UID: "3ba89675-526d-4f5d-b8ff-d676460179fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:53.717120 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:53.717114    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ba89675-526d-4f5d-b8ff-d676460179fd-kube-api-access-8frqq" pod "3ba89675-526d-4f5d-b8ff-d676460179fd" (UID: "3ba89675-526d-4f5d-b8ff-d676460179fd"). InnerVolumeSpecName "kube-api-access-8frqq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:53.817126 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:53.817096    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ba89675-526d-4f5d-b8ff-d676460179fd-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:53.817126 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:53.817121    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8frqq\" (UniqueName: \"kubernetes.io/projected/3ba89675-526d-4f5d-b8ff-d676460179fd-kube-api-access-8frqq\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:08:54.465804 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:54.465776    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9d4043-927m5"
Oct 30 22:08:54.466008 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:54.465800    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9d4043-927m5" event={"ID":"3ba89675-526d-4f5d-b8ff-d676460179fd","Type":"ContainerDied","Data":"6434683688a26389b34f6a92e61f7cca293eb97ad0bb4f6bdfe422b446a2ad25"}
Oct 30 22:08:54.466008 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:08:54.465852    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6434683688a26389b34f6a92e61f7cca293eb97ad0bb4f6bdfe422b446a2ad25"
Oct 30 22:09:02.736632 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:02.736601    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst"]
Oct 30 22:09:02.737269 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:02.737253    5087 cpu_manager.go:414] "containerMap: removing container" podUID="3ba89675-526d-4f5d-b8ff-d676460179fd" containerName="revoke-keys"
Oct 30 22:09:02.737269 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:02.737270    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ba89675-526d-4f5d-b8ff-d676460179fd" containerName="revoke-keys"
Oct 30 22:09:02.737421 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:02.737410    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ba89675-526d-4f5d-b8ff-d676460179fd" containerName="revoke-keys"
Oct 30 22:09:02.741277 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:02.741258    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst"
Oct 30 22:09:02.800453 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:02.800417    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-307368-serving-cert\""
Oct 30 22:09:02.839917 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:02.839876    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vfxw\" (UniqueName: \"kubernetes.io/projected/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-kube-api-access-4vfxw\") pod \"maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst\" (UID: \"c581dd8e-e662-4d4e-9ea8-21d19e46fcf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst"
Oct 30 22:09:02.840176 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:02.840016    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-maas-api-tls\") pod \"maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst\" (UID: \"c581dd8e-e662-4d4e-9ea8-21d19e46fcf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst"
Oct 30 22:09:02.840299 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:02.840181    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-openshift-service-ca\") pod \"maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst\" (UID: \"c581dd8e-e662-4d4e-9ea8-21d19e46fcf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst"
Oct 30 22:09:02.840379 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:02.840301    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-metrics-certs\") pod \"maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst\" (UID: \"c581dd8e-e662-4d4e-9ea8-21d19e46fcf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst"
Oct 30 22:09:02.855681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:02.855649    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst"]
Oct 30 22:09:02.941746 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:02.941709    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-openshift-service-ca\") pod \"maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst\" (UID: \"c581dd8e-e662-4d4e-9ea8-21d19e46fcf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst"
Oct 30 22:09:02.941951 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:02.941775    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-metrics-certs\") pod \"maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst\" (UID: \"c581dd8e-e662-4d4e-9ea8-21d19e46fcf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst"
Oct 30 22:09:02.941951 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:02.941918    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vfxw\" (UniqueName: \"kubernetes.io/projected/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-kube-api-access-4vfxw\") pod \"maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst\" (UID: \"c581dd8e-e662-4d4e-9ea8-21d19e46fcf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst"
Oct 30 22:09:02.942082 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:02.941994    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-maas-api-tls\") pod \"maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst\" (UID: \"c581dd8e-e662-4d4e-9ea8-21d19e46fcf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst"
Oct 30 22:09:02.942143 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:09:02.942118    5087 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-307368-serving-cert: secret "maas-api-e2e-api-b-307368-serving-cert" not found
Oct 30 22:09:02.942237 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:09:02.942222    5087 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-maas-api-tls podName:c581dd8e-e662-4d4e-9ea8-21d19e46fcf0 nodeName:}" failed. No retries permitted until 2026-10-30 22:09:03.442200993 +0000 UTC m=+2137.916704968 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-maas-api-tls") pod "maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst" (UID: "c581dd8e-e662-4d4e-9ea8-21d19e46fcf0") : secret "maas-api-e2e-api-b-307368-serving-cert" not found
Oct 30 22:09:02.942610 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:02.942588    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-openshift-service-ca\") pod \"maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst\" (UID: \"c581dd8e-e662-4d4e-9ea8-21d19e46fcf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst"
Oct 30 22:09:02.944548 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:02.944528    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-metrics-certs\") pod \"maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst\" (UID: \"c581dd8e-e662-4d4e-9ea8-21d19e46fcf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst"
Oct 30 22:09:03.018413 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:03.011087    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vfxw\" (UniqueName: \"kubernetes.io/projected/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-kube-api-access-4vfxw\") pod \"maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst\" (UID: \"c581dd8e-e662-4d4e-9ea8-21d19e46fcf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst"
Oct 30 22:09:03.449044 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:03.448951    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-maas-api-tls\") pod \"maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst\" (UID: \"c581dd8e-e662-4d4e-9ea8-21d19e46fcf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst"
Oct 30 22:09:03.452325 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:03.452292    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-maas-api-tls\") pod \"maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst\" (UID: \"c581dd8e-e662-4d4e-9ea8-21d19e46fcf0\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst"
Oct 30 22:09:03.655939 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:03.655911    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst"
Oct 30 22:09:04.037944 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:04.037912    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst"]
Oct 30 22:09:04.039444 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:09:04.039410    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc581dd8e_e662_4d4e_9ea8_21d19e46fcf0.slice/crio-98bd1e9165ed0f0b1ea8b488e17e03e7bf714bbe750a3295355f3687d2bc15b1 WatchSource:0}: Error finding container 98bd1e9165ed0f0b1ea8b488e17e03e7bf714bbe750a3295355f3687d2bc15b1: Status 404 returned error can't find the container with id 98bd1e9165ed0f0b1ea8b488e17e03e7bf714bbe750a3295355f3687d2bc15b1
Oct 30 22:09:04.540616 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:04.540564    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst" event={"ID":"c581dd8e-e662-4d4e-9ea8-21d19e46fcf0","Type":"ContainerStarted","Data":"98bd1e9165ed0f0b1ea8b488e17e03e7bf714bbe750a3295355f3687d2bc15b1"}
Oct 30 22:09:04.909820 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:04.909793    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29"]
Oct 30 22:09:04.914372 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:04.914324    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29"
Oct 30 22:09:04.968688 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:04.968657    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29"]
Oct 30 22:09:04.999072 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:04.999041    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:09:05.013802 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:05.013776    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:09:05.071582 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:05.071551    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldbt2\" (UniqueName: \"kubernetes.io/projected/67c150f1-88ba-4010-b3ea-750b1cc9db58-kube-api-access-ldbt2\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29\" (UID: \"67c150f1-88ba-4010-b3ea-750b1cc9db58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29"
Oct 30 22:09:05.072098 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:05.071736    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c150f1-88ba-4010-b3ea-750b1cc9db58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29\" (UID: \"67c150f1-88ba-4010-b3ea-750b1cc9db58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29"
Oct 30 22:09:05.072098 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:05.071789    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c150f1-88ba-4010-b3ea-750b1cc9db58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29\" (UID: \"67c150f1-88ba-4010-b3ea-750b1cc9db58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29"
Oct 30 22:09:05.173888 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:05.173843    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c150f1-88ba-4010-b3ea-750b1cc9db58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29\" (UID: \"67c150f1-88ba-4010-b3ea-750b1cc9db58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29"
Oct 30 22:09:05.174090 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:05.173925    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c150f1-88ba-4010-b3ea-750b1cc9db58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29\" (UID: \"67c150f1-88ba-4010-b3ea-750b1cc9db58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29"
Oct 30 22:09:05.174152 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:05.174088    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldbt2\" (UniqueName: \"kubernetes.io/projected/67c150f1-88ba-4010-b3ea-750b1cc9db58-kube-api-access-ldbt2\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29\" (UID: \"67c150f1-88ba-4010-b3ea-750b1cc9db58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29"
Oct 30 22:09:05.175341 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:05.175306    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c150f1-88ba-4010-b3ea-750b1cc9db58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29\" (UID: \"67c150f1-88ba-4010-b3ea-750b1cc9db58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29"
Oct 30 22:09:05.185678 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:05.185647    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c150f1-88ba-4010-b3ea-750b1cc9db58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29\" (UID: \"67c150f1-88ba-4010-b3ea-750b1cc9db58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29"
Oct 30 22:09:05.233979 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:05.233939    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldbt2\" (UniqueName: \"kubernetes.io/projected/67c150f1-88ba-4010-b3ea-750b1cc9db58-kube-api-access-ldbt2\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29\" (UID: \"67c150f1-88ba-4010-b3ea-750b1cc9db58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29"
Oct 30 22:09:05.529274 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:05.529240    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29"
Oct 30 22:09:05.561590 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:05.561543    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst" event={"ID":"c581dd8e-e662-4d4e-9ea8-21d19e46fcf0","Type":"ContainerStarted","Data":"211928ee6cce15b0182af607b9f0903a9a13bc6b9b7ca4693a0b856d671123d3"}
Oct 30 22:09:05.562030 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:05.561969    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst"
Oct 30 22:09:05.678803 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:05.678630    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst" podStartSLOduration=2.897950563 podStartE2EDuration="3.678593196s" totalImagesPullingTime="780.642633ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:05.677938405 +0000 UTC m=+2140.152442392" watchObservedRunningTime="2026-10-30 22:09:05.678593196 +0000 UTC m=+2140.153097177"
Oct 30 22:09:05.915283 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:05.915254    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29"]
Oct 30 22:09:05.916671 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:09:05.916649    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67c150f1_88ba_4010_b3ea_750b1cc9db58.slice/crio-6f76202d64cf69696e7520f1fd79d92bf8340b19a7bed927cd99a2ef1e6c3f47 WatchSource:0}: Error finding container 6f76202d64cf69696e7520f1fd79d92bf8340b19a7bed927cd99a2ef1e6c3f47: Status 404 returned error can't find the container with id 6f76202d64cf69696e7520f1fd79d92bf8340b19a7bed927cd99a2ef1e6c3f47
Oct 30 22:09:06.571050 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:06.570950    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29" event={"ID":"67c150f1-88ba-4010-b3ea-750b1cc9db58","Type":"ContainerStarted","Data":"ff8a49b8661b2f37638305ae5f35e5b2c82fc37011f2e23d4ac005a2f4e83b18"}
Oct 30 22:09:06.571050 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:06.571020    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29" event={"ID":"67c150f1-88ba-4010-b3ea-750b1cc9db58","Type":"ContainerStarted","Data":"6f76202d64cf69696e7520f1fd79d92bf8340b19a7bed927cd99a2ef1e6c3f47"}
Oct 30 22:09:06.688153 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:06.688082    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29" podStartSLOduration=2.688067169 podStartE2EDuration="2.688067169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:06.685097329 +0000 UTC m=+2141.159601464" watchObservedRunningTime="2026-10-30 22:09:06.688067169 +0000 UTC m=+2141.162571148"
Oct 30 22:09:07.582617 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:07.582576    5087 generic.go:356] "Generic (PLEG): container finished" podID="67c150f1-88ba-4010-b3ea-750b1cc9db58" containerID="ff8a49b8661b2f37638305ae5f35e5b2c82fc37011f2e23d4ac005a2f4e83b18" exitCode=0
Oct 30 22:09:07.583200 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:07.583168    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29" event={"ID":"67c150f1-88ba-4010-b3ea-750b1cc9db58","Type":"ContainerDied","Data":"ff8a49b8661b2f37638305ae5f35e5b2c82fc37011f2e23d4ac005a2f4e83b18"}
Oct 30 22:09:08.776223 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:08.776196    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29"
Oct 30 22:09:08.926245 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:08.926165    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67c150f1-88ba-4010-b3ea-750b1cc9db58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c150f1-88ba-4010-b3ea-750b1cc9db58-maas-api-openshift-ca\") pod \"67c150f1-88ba-4010-b3ea-750b1cc9db58\" (UID: \"67c150f1-88ba-4010-b3ea-750b1cc9db58\") "
Oct 30 22:09:08.926245 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:08.926199    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67c150f1-88ba-4010-b3ea-750b1cc9db58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c150f1-88ba-4010-b3ea-750b1cc9db58-maas-api-xks-ca\") pod \"67c150f1-88ba-4010-b3ea-750b1cc9db58\" (UID: \"67c150f1-88ba-4010-b3ea-750b1cc9db58\") "
Oct 30 22:09:08.926472 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:08.926461    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67c150f1-88ba-4010-b3ea-750b1cc9db58-kube-api-access-ldbt2\" (UniqueName: \"kubernetes.io/projected/67c150f1-88ba-4010-b3ea-750b1cc9db58-kube-api-access-ldbt2\") pod \"67c150f1-88ba-4010-b3ea-750b1cc9db58\" (UID: \"67c150f1-88ba-4010-b3ea-750b1cc9db58\") "
Oct 30 22:09:08.926637 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:08.926604    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67c150f1-88ba-4010-b3ea-750b1cc9db58-maas-api-openshift-ca" pod "67c150f1-88ba-4010-b3ea-750b1cc9db58" (UID: "67c150f1-88ba-4010-b3ea-750b1cc9db58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:08.926983 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:08.926957    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67c150f1-88ba-4010-b3ea-750b1cc9db58-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:08.928701 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:08.928671    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67c150f1-88ba-4010-b3ea-750b1cc9db58-maas-api-xks-ca" pod "67c150f1-88ba-4010-b3ea-750b1cc9db58" (UID: "67c150f1-88ba-4010-b3ea-750b1cc9db58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:08.928890 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:08.928866    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c150f1-88ba-4010-b3ea-750b1cc9db58-kube-api-access-ldbt2" pod "67c150f1-88ba-4010-b3ea-750b1cc9db58" (UID: "67c150f1-88ba-4010-b3ea-750b1cc9db58"). InnerVolumeSpecName "kube-api-access-ldbt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:09.027824 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:09.027788    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldbt2\" (UniqueName: \"kubernetes.io/projected/67c150f1-88ba-4010-b3ea-750b1cc9db58-kube-api-access-ldbt2\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:09.027824 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:09.027819    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67c150f1-88ba-4010-b3ea-750b1cc9db58-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:09.599152 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:09.599119    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29"
Oct 30 22:09:09.599342 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:09.599150    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29" event={"ID":"67c150f1-88ba-4010-b3ea-750b1cc9db58","Type":"ContainerDied","Data":"6f76202d64cf69696e7520f1fd79d92bf8340b19a7bed927cd99a2ef1e6c3f47"}
Oct 30 22:09:09.599342 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:09.599229    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f76202d64cf69696e7520f1fd79d92bf8340b19a7bed927cd99a2ef1e6c3f47"
Oct 30 22:09:11.580083 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:11.580052    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst"
Oct 30 22:09:17.167808 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:09:17.167774    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:09:20.429332 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:20.429293    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-99f69c-5pvr7"]
Oct 30 22:09:20.478200 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:20.478164    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-99f69c-5pvr7"]
Oct 30 22:09:21.712734 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:21.712695    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2f5bf30-e073-4b5a-be6f-48d0e07754d6" path="/var/lib/kubelet/pods/f2f5bf30-e073-4b5a-be6f-48d0e07754d6/volumes"
Oct 30 22:09:28.165498 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:28.165459    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b"]
Oct 30 22:09:28.166089 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:28.166074    5087 cpu_manager.go:414] "containerMap: removing container" podUID="67c150f1-88ba-4010-b3ea-750b1cc9db58" containerName="revoke-keys"
Oct 30 22:09:28.166089 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:28.166088    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67c150f1-88ba-4010-b3ea-750b1cc9db58" containerName="revoke-keys"
Oct 30 22:09:28.166220 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:28.166180    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="67c150f1-88ba-4010-b3ea-750b1cc9db58" containerName="revoke-keys"
Oct 30 22:09:28.170840 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:28.170820    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b"
Oct 30 22:09:28.220858 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:28.220808    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07358162-aa30-4455-9759-1e7fcf1c4871-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b\" (UID: \"07358162-aa30-4455-9759-1e7fcf1c4871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b"
Oct 30 22:09:28.221027 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:28.220880    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4fbj\" (UniqueName: \"kubernetes.io/projected/07358162-aa30-4455-9759-1e7fcf1c4871-kube-api-access-z4fbj\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b\" (UID: \"07358162-aa30-4455-9759-1e7fcf1c4871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b"
Oct 30 22:09:28.221073 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:28.221044    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07358162-aa30-4455-9759-1e7fcf1c4871-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b\" (UID: \"07358162-aa30-4455-9759-1e7fcf1c4871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b"
Oct 30 22:09:28.272959 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:28.272925    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:09:28.273213 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:28.272927    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:09:28.278442 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:28.278412    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b"]
Oct 30 22:09:28.322583 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:28.322538    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07358162-aa30-4455-9759-1e7fcf1c4871-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b\" (UID: \"07358162-aa30-4455-9759-1e7fcf1c4871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b"
Oct 30 22:09:28.322774 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:28.322705    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07358162-aa30-4455-9759-1e7fcf1c4871-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b\" (UID: \"07358162-aa30-4455-9759-1e7fcf1c4871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b"
Oct 30 22:09:28.322774 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:28.322745    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4fbj\" (UniqueName: \"kubernetes.io/projected/07358162-aa30-4455-9759-1e7fcf1c4871-kube-api-access-z4fbj\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b\" (UID: \"07358162-aa30-4455-9759-1e7fcf1c4871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b"
Oct 30 22:09:28.323159 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:28.323137    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07358162-aa30-4455-9759-1e7fcf1c4871-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b\" (UID: \"07358162-aa30-4455-9759-1e7fcf1c4871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b"
Oct 30 22:09:28.325341 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:28.325308    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07358162-aa30-4455-9759-1e7fcf1c4871-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b\" (UID: \"07358162-aa30-4455-9759-1e7fcf1c4871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b"
Oct 30 22:09:28.381152 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:28.381110    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4fbj\" (UniqueName: \"kubernetes.io/projected/07358162-aa30-4455-9759-1e7fcf1c4871-kube-api-access-z4fbj\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b\" (UID: \"07358162-aa30-4455-9759-1e7fcf1c4871\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b"
Oct 30 22:09:28.486142 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:28.486106    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b"
Oct 30 22:09:28.869299 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:28.869274    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b"]
Oct 30 22:09:28.871078 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:09:28.871047    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07358162_aa30_4455_9759_1e7fcf1c4871.slice/crio-217358b2feda1ce189f6ef48c9a1fec175addafff8766a99a5dbece86b2066dd WatchSource:0}: Error finding container 217358b2feda1ce189f6ef48c9a1fec175addafff8766a99a5dbece86b2066dd: Status 404 returned error can't find the container with id 217358b2feda1ce189f6ef48c9a1fec175addafff8766a99a5dbece86b2066dd
Oct 30 22:09:29.752333 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:29.752289    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b" event={"ID":"07358162-aa30-4455-9759-1e7fcf1c4871","Type":"ContainerStarted","Data":"9f54313622fcdc7445e121626a9383a51729f49ea1b17024116cf7eba9bddf73"}
Oct 30 22:09:29.752333 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:29.752365    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b" event={"ID":"07358162-aa30-4455-9759-1e7fcf1c4871","Type":"ContainerStarted","Data":"217358b2feda1ce189f6ef48c9a1fec175addafff8766a99a5dbece86b2066dd"}
Oct 30 22:09:29.870817 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:29.870770    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b" podStartSLOduration=1.870756774 podStartE2EDuration="1.870756774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:29.866879907 +0000 UTC m=+2164.341383886" watchObservedRunningTime="2026-10-30 22:09:29.870756774 +0000 UTC m=+2164.345260789"
Oct 30 22:09:30.760958 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:30.760923    5087 generic.go:356] "Generic (PLEG): container finished" podID="07358162-aa30-4455-9759-1e7fcf1c4871" containerID="9f54313622fcdc7445e121626a9383a51729f49ea1b17024116cf7eba9bddf73" exitCode=0
Oct 30 22:09:30.761406 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:30.760968    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b" event={"ID":"07358162-aa30-4455-9759-1e7fcf1c4871","Type":"ContainerDied","Data":"9f54313622fcdc7445e121626a9383a51729f49ea1b17024116cf7eba9bddf73"}
Oct 30 22:09:31.931882 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:31.931859    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b"
Oct 30 22:09:32.073229 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:32.073140    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07358162-aa30-4455-9759-1e7fcf1c4871-kube-api-access-z4fbj\" (UniqueName: \"kubernetes.io/projected/07358162-aa30-4455-9759-1e7fcf1c4871-kube-api-access-z4fbj\") pod \"07358162-aa30-4455-9759-1e7fcf1c4871\" (UID: \"07358162-aa30-4455-9759-1e7fcf1c4871\") "
Oct 30 22:09:32.073229 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:32.073212    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07358162-aa30-4455-9759-1e7fcf1c4871-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07358162-aa30-4455-9759-1e7fcf1c4871-maas-api-openshift-ca\") pod \"07358162-aa30-4455-9759-1e7fcf1c4871\" (UID: \"07358162-aa30-4455-9759-1e7fcf1c4871\") "
Oct 30 22:09:32.073445 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:32.073367    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07358162-aa30-4455-9759-1e7fcf1c4871-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07358162-aa30-4455-9759-1e7fcf1c4871-maas-api-xks-ca\") pod \"07358162-aa30-4455-9759-1e7fcf1c4871\" (UID: \"07358162-aa30-4455-9759-1e7fcf1c4871\") "
Oct 30 22:09:32.074620 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:32.074586    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07358162-aa30-4455-9759-1e7fcf1c4871-maas-api-openshift-ca" pod "07358162-aa30-4455-9759-1e7fcf1c4871" (UID: "07358162-aa30-4455-9759-1e7fcf1c4871"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:32.076857 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:32.076830    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07358162-aa30-4455-9759-1e7fcf1c4871-kube-api-access-z4fbj" pod "07358162-aa30-4455-9759-1e7fcf1c4871" (UID: "07358162-aa30-4455-9759-1e7fcf1c4871"). InnerVolumeSpecName "kube-api-access-z4fbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:32.078099 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:32.078071    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07358162-aa30-4455-9759-1e7fcf1c4871-maas-api-xks-ca" pod "07358162-aa30-4455-9759-1e7fcf1c4871" (UID: "07358162-aa30-4455-9759-1e7fcf1c4871"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:32.175158 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:32.175127    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4fbj\" (UniqueName: \"kubernetes.io/projected/07358162-aa30-4455-9759-1e7fcf1c4871-kube-api-access-z4fbj\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:32.175158 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:32.175156    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07358162-aa30-4455-9759-1e7fcf1c4871-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:32.175411 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:32.175167    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07358162-aa30-4455-9759-1e7fcf1c4871-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:32.777484 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:32.777458    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b"
Oct 30 22:09:32.777670 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:32.777496    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b" event={"ID":"07358162-aa30-4455-9759-1e7fcf1c4871","Type":"ContainerDied","Data":"217358b2feda1ce189f6ef48c9a1fec175addafff8766a99a5dbece86b2066dd"}
Oct 30 22:09:32.777670 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:32.777547    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="217358b2feda1ce189f6ef48c9a1fec175addafff8766a99a5dbece86b2066dd"
Oct 30 22:09:33.696584 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:33.696549    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"]
Oct 30 22:09:33.697091 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:33.696726    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc" podUID="dafd13b7-c191-4d83-b94c-371bf53a9fe5" containerName="istio-proxy" containerID="cri-o://7c16092a1b86edc469898c83c4d35c267aa765c3ceb5b2a3c0171b29ef51687a" gracePeriod=30
Oct 30 22:09:34.554817 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:34.554780    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c030d4a4-k4mhh"]
Oct 30 22:09:34.555546 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:34.555525    5087 cpu_manager.go:414] "containerMap: removing container" podUID="07358162-aa30-4455-9759-1e7fcf1c4871" containerName="revoke-keys"
Oct 30 22:09:34.555546 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:34.555547    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07358162-aa30-4455-9759-1e7fcf1c4871" containerName="revoke-keys"
Oct 30 22:09:34.555748 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:34.555685    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="07358162-aa30-4455-9759-1e7fcf1c4871" containerName="revoke-keys"
Oct 30 22:09:34.560020 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:34.559996    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c030d4a4-k4mhh"
Oct 30 22:09:34.614947 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:34.614916    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c030d4a4-k4mhh"]
Oct 30 22:09:34.659151 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:34.659119    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:09:34.659319 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:34.659162    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:09:34.707778 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:34.707747    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6825e7-85c9-4cba-8f6e-e52669c32616-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c030d4a4-k4mhh\" (UID: \"9e6825e7-85c9-4cba-8f6e-e52669c32616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c030d4a4-k4mhh"
Oct 30 22:09:34.708140 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:34.707938    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg84d\" (UniqueName: \"kubernetes.io/projected/9e6825e7-85c9-4cba-8f6e-e52669c32616-kube-api-access-vg84d\") pod \"maas-api-revoke-sub-c030d4a4-k4mhh\" (UID: \"9e6825e7-85c9-4cba-8f6e-e52669c32616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c030d4a4-k4mhh"
Oct 30 22:09:34.708140 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:34.707974    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6825e7-85c9-4cba-8f6e-e52669c32616-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c030d4a4-k4mhh\" (UID: \"9e6825e7-85c9-4cba-8f6e-e52669c32616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c030d4a4-k4mhh"
Oct 30 22:09:34.809388 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:34.809294    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg84d\" (UniqueName: \"kubernetes.io/projected/9e6825e7-85c9-4cba-8f6e-e52669c32616-kube-api-access-vg84d\") pod \"maas-api-revoke-sub-c030d4a4-k4mhh\" (UID: \"9e6825e7-85c9-4cba-8f6e-e52669c32616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c030d4a4-k4mhh"
Oct 30 22:09:34.809388 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:34.809334    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6825e7-85c9-4cba-8f6e-e52669c32616-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c030d4a4-k4mhh\" (UID: \"9e6825e7-85c9-4cba-8f6e-e52669c32616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c030d4a4-k4mhh"
Oct 30 22:09:34.809388 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:34.809376    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6825e7-85c9-4cba-8f6e-e52669c32616-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c030d4a4-k4mhh\" (UID: \"9e6825e7-85c9-4cba-8f6e-e52669c32616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c030d4a4-k4mhh"
Oct 30 22:09:34.810113 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:34.810090    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6825e7-85c9-4cba-8f6e-e52669c32616-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c030d4a4-k4mhh\" (UID: \"9e6825e7-85c9-4cba-8f6e-e52669c32616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c030d4a4-k4mhh"
Oct 30 22:09:34.811988 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:34.811964    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6825e7-85c9-4cba-8f6e-e52669c32616-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c030d4a4-k4mhh\" (UID: \"9e6825e7-85c9-4cba-8f6e-e52669c32616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c030d4a4-k4mhh"
Oct 30 22:09:34.866819 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:34.866786    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg84d\" (UniqueName: \"kubernetes.io/projected/9e6825e7-85c9-4cba-8f6e-e52669c32616-kube-api-access-vg84d\") pod \"maas-api-revoke-sub-c030d4a4-k4mhh\" (UID: \"9e6825e7-85c9-4cba-8f6e-e52669c32616\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c030d4a4-k4mhh"
Oct 30 22:09:34.877910 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:34.877883    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c030d4a4-k4mhh"
Oct 30 22:09:34.898403 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:34.898329    5087 patch_prober.go:29] interesting pod/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:09:34.898507 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:34.898434    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc" podUID="dafd13b7-c191-4d83-b94c-371bf53a9fe5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:09:35.267883 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:35.267860    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c030d4a4-k4mhh"]
Oct 30 22:09:35.269364 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:09:35.269325    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e6825e7_85c9_4cba_8f6e_e52669c32616.slice/crio-f2d459281ee836e4393c4f0b8049cff745b68c37c07e877b5c68e8239254c4d7 WatchSource:0}: Error finding container f2d459281ee836e4393c4f0b8049cff745b68c37c07e877b5c68e8239254c4d7: Status 404 returned error can't find the container with id f2d459281ee836e4393c4f0b8049cff745b68c37c07e877b5c68e8239254c4d7
Oct 30 22:09:35.800656 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:35.800614    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c030d4a4-k4mhh" event={"ID":"9e6825e7-85c9-4cba-8f6e-e52669c32616","Type":"ContainerStarted","Data":"7961b3c67f4a9b559de92d0df2455b636c15eb6717dbbe2c1e7e7672fea12668"}
Oct 30 22:09:35.801169 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:35.800681    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c030d4a4-k4mhh" event={"ID":"9e6825e7-85c9-4cba-8f6e-e52669c32616","Type":"ContainerStarted","Data":"f2d459281ee836e4393c4f0b8049cff745b68c37c07e877b5c68e8239254c4d7"}
Oct 30 22:09:35.914940 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:35.914843    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c030d4a4-k4mhh" podStartSLOduration=1.9148280519999998 podStartE2EDuration="1.914828052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:35.913051624 +0000 UTC m=+2170.387555604" watchObservedRunningTime="2026-10-30 22:09:35.914828052 +0000 UTC m=+2170.389332089"
Oct 30 22:09:36.808215 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:36.808179    5087 generic.go:356] "Generic (PLEG): container finished" podID="9e6825e7-85c9-4cba-8f6e-e52669c32616" containerID="7961b3c67f4a9b559de92d0df2455b636c15eb6717dbbe2c1e7e7672fea12668" exitCode=0
Oct 30 22:09:36.808708 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:36.808269    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c030d4a4-k4mhh" event={"ID":"9e6825e7-85c9-4cba-8f6e-e52669c32616","Type":"ContainerDied","Data":"7961b3c67f4a9b559de92d0df2455b636c15eb6717dbbe2c1e7e7672fea12668"}
Oct 30 22:09:37.993783 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:37.993756    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c030d4a4-k4mhh"
Oct 30 22:09:38.153995 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:38.153913    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e6825e7-85c9-4cba-8f6e-e52669c32616-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6825e7-85c9-4cba-8f6e-e52669c32616-maas-api-xks-ca\") pod \"9e6825e7-85c9-4cba-8f6e-e52669c32616\" (UID: \"9e6825e7-85c9-4cba-8f6e-e52669c32616\") "
Oct 30 22:09:38.154165 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:38.154075    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e6825e7-85c9-4cba-8f6e-e52669c32616-kube-api-access-vg84d\" (UniqueName: \"kubernetes.io/projected/9e6825e7-85c9-4cba-8f6e-e52669c32616-kube-api-access-vg84d\") pod \"9e6825e7-85c9-4cba-8f6e-e52669c32616\" (UID: \"9e6825e7-85c9-4cba-8f6e-e52669c32616\") "
Oct 30 22:09:38.154165 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:38.154123    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e6825e7-85c9-4cba-8f6e-e52669c32616-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6825e7-85c9-4cba-8f6e-e52669c32616-maas-api-openshift-ca\") pod \"9e6825e7-85c9-4cba-8f6e-e52669c32616\" (UID: \"9e6825e7-85c9-4cba-8f6e-e52669c32616\") "
Oct 30 22:09:38.155247 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:38.155214    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e6825e7-85c9-4cba-8f6e-e52669c32616-maas-api-openshift-ca" pod "9e6825e7-85c9-4cba-8f6e-e52669c32616" (UID: "9e6825e7-85c9-4cba-8f6e-e52669c32616"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:38.156674 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:38.156645    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e6825e7-85c9-4cba-8f6e-e52669c32616-maas-api-xks-ca" pod "9e6825e7-85c9-4cba-8f6e-e52669c32616" (UID: "9e6825e7-85c9-4cba-8f6e-e52669c32616"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:38.156797 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:38.156778    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e6825e7-85c9-4cba-8f6e-e52669c32616-kube-api-access-vg84d" pod "9e6825e7-85c9-4cba-8f6e-e52669c32616" (UID: "9e6825e7-85c9-4cba-8f6e-e52669c32616"). InnerVolumeSpecName "kube-api-access-vg84d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:38.256783 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:38.256743    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg84d\" (UniqueName: \"kubernetes.io/projected/9e6825e7-85c9-4cba-8f6e-e52669c32616-kube-api-access-vg84d\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:38.256783 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:38.256779    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e6825e7-85c9-4cba-8f6e-e52669c32616-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:38.257011 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:38.256796    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e6825e7-85c9-4cba-8f6e-e52669c32616-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:38.579089 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:38.579055    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-485d1e-66c964c857-fddzn"]
Oct 30 22:09:38.689948 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:38.689903    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-485d1e-58bccc79c9-jk9hv"]
Oct 30 22:09:38.822590 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:38.822557    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c030d4a4-k4mhh" event={"ID":"9e6825e7-85c9-4cba-8f6e-e52669c32616","Type":"ContainerDied","Data":"f2d459281ee836e4393c4f0b8049cff745b68c37c07e877b5c68e8239254c4d7"}
Oct 30 22:09:38.822719 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:38.822612    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2d459281ee836e4393c4f0b8049cff745b68c37c07e877b5c68e8239254c4d7"
Oct 30 22:09:38.822719 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:38.822574    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c030d4a4-k4mhh"
Oct 30 22:09:38.824489 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:38.824466    5087 generic.go:356] "Generic (PLEG): container finished" podID="dafd13b7-c191-4d83-b94c-371bf53a9fe5" containerID="7c16092a1b86edc469898c83c4d35c267aa765c3ceb5b2a3c0171b29ef51687a" exitCode=0
Oct 30 22:09:38.824612 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:38.824538    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc" event={"ID":"dafd13b7-c191-4d83-b94c-371bf53a9fe5","Type":"ContainerDied","Data":"7c16092a1b86edc469898c83c4d35c267aa765c3ceb5b2a3c0171b29ef51687a"}
Oct 30 22:09:39.064200 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.064177    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 22:09:39.175245 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.175155    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-workload-certs\") pod \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") "
Oct 30 22:09:39.175245 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.175228    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dafd13b7-c191-4d83-b94c-371bf53a9fe5-kube-api-access-bjnw4\" (UniqueName: \"kubernetes.io/projected/dafd13b7-c191-4d83-b94c-371bf53a9fe5-kube-api-access-bjnw4\") pod \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") "
Oct 30 22:09:39.175540 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.175284    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-credential-socket\") pod \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") "
Oct 30 22:09:39.175540 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.175315    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-workload-socket\") pod \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") "
Oct 30 22:09:39.175540 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.175373    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istio-envoy\") pod \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") "
Oct 30 22:09:39.175540 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.175457    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istio-podinfo\") pod \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") "
Oct 30 22:09:39.175540 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.175494    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istio-data\") pod \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") "
Oct 30 22:09:39.175811 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.175538    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-workload-certs" pod "dafd13b7-c191-4d83-b94c-371bf53a9fe5" (UID: "dafd13b7-c191-4d83-b94c-371bf53a9fe5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:39.175811 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.175616    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istiod-ca-cert\") pod \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") "
Oct 30 22:09:39.175811 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.175678    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istio-token\" (UniqueName: \"kubernetes.io/projected/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istio-token\") pod \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\" (UID: \"dafd13b7-c191-4d83-b94c-371bf53a9fe5\") "
Oct 30 22:09:39.175811 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.175773    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istio-data" pod "dafd13b7-c191-4d83-b94c-371bf53a9fe5" (UID: "dafd13b7-c191-4d83-b94c-371bf53a9fe5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:39.176000 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.175981    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istiod-ca-cert" pod "dafd13b7-c191-4d83-b94c-371bf53a9fe5" (UID: "dafd13b7-c191-4d83-b94c-371bf53a9fe5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:39.176062 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.176000    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-workload-socket" pod "dafd13b7-c191-4d83-b94c-371bf53a9fe5" (UID: "dafd13b7-c191-4d83-b94c-371bf53a9fe5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:39.176557 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.176479    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:39.176557 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.176509    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:39.176557 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.176524    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:39.176557 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.176539    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:39.176557 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.176619    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-credential-socket" pod "dafd13b7-c191-4d83-b94c-371bf53a9fe5" (UID: "dafd13b7-c191-4d83-b94c-371bf53a9fe5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:39.178307 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.178273    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istio-envoy" pod "dafd13b7-c191-4d83-b94c-371bf53a9fe5" (UID: "dafd13b7-c191-4d83-b94c-371bf53a9fe5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:39.178467 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.178316    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istio-token" pod "dafd13b7-c191-4d83-b94c-371bf53a9fe5" (UID: "dafd13b7-c191-4d83-b94c-371bf53a9fe5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:39.178467 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.178341    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dafd13b7-c191-4d83-b94c-371bf53a9fe5-kube-api-access-bjnw4" pod "dafd13b7-c191-4d83-b94c-371bf53a9fe5" (UID: "dafd13b7-c191-4d83-b94c-371bf53a9fe5"). InnerVolumeSpecName "kube-api-access-bjnw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:39.179060 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.179036    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istio-podinfo" pod "dafd13b7-c191-4d83-b94c-371bf53a9fe5" (UID: "dafd13b7-c191-4d83-b94c-371bf53a9fe5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:09:39.278128 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.278089    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:39.278128 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.278131    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjnw4\" (UniqueName: \"kubernetes.io/projected/dafd13b7-c191-4d83-b94c-371bf53a9fe5-kube-api-access-bjnw4\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:39.278338 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.278147    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:39.278338 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.278161    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:39.278338 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.278175    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dafd13b7-c191-4d83-b94c-371bf53a9fe5-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:39.832524 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.832490    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"
Oct 30 22:09:39.832667 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.832523    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc" event={"ID":"dafd13b7-c191-4d83-b94c-371bf53a9fe5","Type":"ContainerDied","Data":"777860780e80cc85966481c69fd5d53755258cb3e476a8fb334230067a9b7bac"}
Oct 30 22:09:39.832667 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.832586    5087 scope.go:122] "RemoveContainer" containerID="7c16092a1b86edc469898c83c4d35c267aa765c3ceb5b2a3c0171b29ef51687a"
Oct 30 22:09:39.995899 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:39.995867    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"]
Oct 30 22:09:40.055662 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:40.055617    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-557708-openshift-default-bd7dbddd5-fj8mc"]
Oct 30 22:09:41.177558 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:41.177513    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c8c4d-mdhnv"]
Oct 30 22:09:41.178145 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:41.178130    5087 cpu_manager.go:414] "containerMap: removing container" podUID="dafd13b7-c191-4d83-b94c-371bf53a9fe5" containerName="istio-proxy"
Oct 30 22:09:41.178145 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:41.178145    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dafd13b7-c191-4d83-b94c-371bf53a9fe5" containerName="istio-proxy"
Oct 30 22:09:41.178271 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:41.178164    5087 cpu_manager.go:414] "containerMap: removing container" podUID="9e6825e7-85c9-4cba-8f6e-e52669c32616" containerName="revoke-keys"
Oct 30 22:09:41.178271 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:41.178170    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e6825e7-85c9-4cba-8f6e-e52669c32616" containerName="revoke-keys"
Oct 30 22:09:41.178271 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:41.178263    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e6825e7-85c9-4cba-8f6e-e52669c32616" containerName="revoke-keys"
Oct 30 22:09:41.178461 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:41.178278    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="dafd13b7-c191-4d83-b94c-371bf53a9fe5" containerName="istio-proxy"
Oct 30 22:09:41.183245 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:41.183224    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c8c4d-mdhnv"
Oct 30 22:09:41.239225 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:41.239196    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c8c4d-mdhnv"]
Oct 30 22:09:41.284146 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:41.284112    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:09:41.284336 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:41.284184    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:09:41.304046 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:41.304018    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5801207-d12c-435e-b488-362e01e8c6d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9c8c4d-mdhnv\" (UID: \"b5801207-d12c-435e-b488-362e01e8c6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c8c4d-mdhnv"
Oct 30 22:09:41.304162 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:41.304123    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp7z9\" (UniqueName: \"kubernetes.io/projected/b5801207-d12c-435e-b488-362e01e8c6d4-kube-api-access-bp7z9\") pod \"maas-api-revoke-keys-e2e-api-a-9c8c4d-mdhnv\" (UID: \"b5801207-d12c-435e-b488-362e01e8c6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c8c4d-mdhnv"
Oct 30 22:09:41.304240 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:41.304179    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5801207-d12c-435e-b488-362e01e8c6d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9c8c4d-mdhnv\" (UID: \"b5801207-d12c-435e-b488-362e01e8c6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c8c4d-mdhnv"
Oct 30 22:09:41.405068 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:41.405040    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp7z9\" (UniqueName: \"kubernetes.io/projected/b5801207-d12c-435e-b488-362e01e8c6d4-kube-api-access-bp7z9\") pod \"maas-api-revoke-keys-e2e-api-a-9c8c4d-mdhnv\" (UID: \"b5801207-d12c-435e-b488-362e01e8c6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c8c4d-mdhnv"
Oct 30 22:09:41.405239 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:41.405102    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5801207-d12c-435e-b488-362e01e8c6d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9c8c4d-mdhnv\" (UID: \"b5801207-d12c-435e-b488-362e01e8c6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c8c4d-mdhnv"
Oct 30 22:09:41.405432 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:41.405248    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5801207-d12c-435e-b488-362e01e8c6d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9c8c4d-mdhnv\" (UID: \"b5801207-d12c-435e-b488-362e01e8c6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c8c4d-mdhnv"
Oct 30 22:09:41.406084 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:41.405980    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5801207-d12c-435e-b488-362e01e8c6d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9c8c4d-mdhnv\" (UID: \"b5801207-d12c-435e-b488-362e01e8c6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c8c4d-mdhnv"
Oct 30 22:09:41.407727 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:41.407698    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5801207-d12c-435e-b488-362e01e8c6d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9c8c4d-mdhnv\" (UID: \"b5801207-d12c-435e-b488-362e01e8c6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c8c4d-mdhnv"
Oct 30 22:09:41.461994 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:41.461920    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp7z9\" (UniqueName: \"kubernetes.io/projected/b5801207-d12c-435e-b488-362e01e8c6d4-kube-api-access-bp7z9\") pod \"maas-api-revoke-keys-e2e-api-a-9c8c4d-mdhnv\" (UID: \"b5801207-d12c-435e-b488-362e01e8c6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c8c4d-mdhnv"
Oct 30 22:09:41.498526 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:41.498504    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c8c4d-mdhnv"
Oct 30 22:09:41.722159 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:41.722064    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dafd13b7-c191-4d83-b94c-371bf53a9fe5" path="/var/lib/kubelet/pods/dafd13b7-c191-4d83-b94c-371bf53a9fe5/volumes"
Oct 30 22:09:41.888847 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:41.888819    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c8c4d-mdhnv"]
Oct 30 22:09:41.890676 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:09:41.890645    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5801207_d12c_435e_b488_362e01e8c6d4.slice/crio-05f5863663c638ada514470e41a6abc2a688785e154117ed5dd1b36771e72f68 WatchSource:0}: Error finding container 05f5863663c638ada514470e41a6abc2a688785e154117ed5dd1b36771e72f68: Status 404 returned error can't find the container with id 05f5863663c638ada514470e41a6abc2a688785e154117ed5dd1b36771e72f68
Oct 30 22:09:42.858260 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:42.858230    5087 generic.go:356] "Generic (PLEG): container finished" podID="b5801207-d12c-435e-b488-362e01e8c6d4" containerID="6fb84d79935898195cc4a743e0d535fb7ab81832e6d3e9866d76d40f261a504c" exitCode=0
Oct 30 22:09:42.858653 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:42.858306    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c8c4d-mdhnv" event={"ID":"b5801207-d12c-435e-b488-362e01e8c6d4","Type":"ContainerDied","Data":"6fb84d79935898195cc4a743e0d535fb7ab81832e6d3e9866d76d40f261a504c"}
Oct 30 22:09:42.858653 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:42.858373    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c8c4d-mdhnv" event={"ID":"b5801207-d12c-435e-b488-362e01e8c6d4","Type":"ContainerStarted","Data":"05f5863663c638ada514470e41a6abc2a688785e154117ed5dd1b36771e72f68"}
Oct 30 22:09:43.579622 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:43.579556    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-485d1e-66c964c857-fddzn" podUID="72563fad-9b54-4005-a4eb-1466011220f8" containerName="payload-processing" containerID="cri-o://8c0e53a81e7fe393befbe0df60af66331c8288d0eca4b93deb949306d4da6b2d" gracePeriod=25
Oct 30 22:09:43.690892 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:43.690849    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-485d1e-58bccc79c9-jk9hv" podUID="94fcd256-faba-490e-9680-d986a9635413" containerName="payload-pre-processing" containerID="cri-o://b156f7d992db23f26da07c58f46ab9dcf72b827dff754a7b06614db837c7a684" gracePeriod=25
Oct 30 22:09:43.866225 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:43.866197    5087 generic.go:356] "Generic (PLEG): container finished" podID="72563fad-9b54-4005-a4eb-1466011220f8" containerID="8c0e53a81e7fe393befbe0df60af66331c8288d0eca4b93deb949306d4da6b2d" exitCode=0
Oct 30 22:09:43.866623 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:43.866329    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-485d1e-66c964c857-fddzn" event={"ID":"72563fad-9b54-4005-a4eb-1466011220f8","Type":"ContainerDied","Data":"8c0e53a81e7fe393befbe0df60af66331c8288d0eca4b93deb949306d4da6b2d"}
Oct 30 22:09:43.866623 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:43.866391    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-485d1e-66c964c857-fddzn" event={"ID":"72563fad-9b54-4005-a4eb-1466011220f8","Type":"ContainerDied","Data":"f8ec9af6c1c4f1014e24757cbdf34273634c3e8c45fe5ee599b7d1f421f38db8"}
Oct 30 22:09:43.866623 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:43.866408    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8ec9af6c1c4f1014e24757cbdf34273634c3e8c45fe5ee599b7d1f421f38db8"
Oct 30 22:09:43.868328 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:43.868298    5087 generic.go:356] "Generic (PLEG): container finished" podID="94fcd256-faba-490e-9680-d986a9635413" containerID="b156f7d992db23f26da07c58f46ab9dcf72b827dff754a7b06614db837c7a684" exitCode=0
Oct 30 22:09:43.868738 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:43.868713    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-485d1e-58bccc79c9-jk9hv" event={"ID":"94fcd256-faba-490e-9680-d986a9635413","Type":"ContainerDied","Data":"b156f7d992db23f26da07c58f46ab9dcf72b827dff754a7b06614db837c7a684"}
Oct 30 22:09:43.910287 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:43.909768    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-485d1e-66c964c857-fddzn"
Oct 30 22:09:43.944410 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:43.942927    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72563fad-9b54-4005-a4eb-1466011220f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72563fad-9b54-4005-a4eb-1466011220f8-plugins-config-volume\") pod \"72563fad-9b54-4005-a4eb-1466011220f8\" (UID: \"72563fad-9b54-4005-a4eb-1466011220f8\") "
Oct 30 22:09:43.944410 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:43.942996    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72563fad-9b54-4005-a4eb-1466011220f8-kube-api-access-66tv6\" (UniqueName: \"kubernetes.io/projected/72563fad-9b54-4005-a4eb-1466011220f8-kube-api-access-66tv6\") pod \"72563fad-9b54-4005-a4eb-1466011220f8\" (UID: \"72563fad-9b54-4005-a4eb-1466011220f8\") "
Oct 30 22:09:43.944410 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:43.943335    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72563fad-9b54-4005-a4eb-1466011220f8-plugins-config-volume" pod "72563fad-9b54-4005-a4eb-1466011220f8" (UID: "72563fad-9b54-4005-a4eb-1466011220f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:43.944410 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:43.943945    5087 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72563fad-9b54-4005-a4eb-1466011220f8-plugins-config-volume\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:43.951820 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:43.951666    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72563fad-9b54-4005-a4eb-1466011220f8-kube-api-access-66tv6" pod "72563fad-9b54-4005-a4eb-1466011220f8" (UID: "72563fad-9b54-4005-a4eb-1466011220f8"). InnerVolumeSpecName "kube-api-access-66tv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:44.010272 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:44.010244    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-485d1e-58bccc79c9-jk9hv"
Oct 30 22:09:44.044611 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:44.044574    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c8c4d-mdhnv"
Oct 30 22:09:44.045055 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:44.045015    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94fcd256-faba-490e-9680-d986a9635413-kube-api-access-chlq8\" (UniqueName: \"kubernetes.io/projected/94fcd256-faba-490e-9680-d986a9635413-kube-api-access-chlq8\") pod \"94fcd256-faba-490e-9680-d986a9635413\" (UID: \"94fcd256-faba-490e-9680-d986a9635413\") "
Oct 30 22:09:44.045166 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:44.045087    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94fcd256-faba-490e-9680-d986a9635413-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94fcd256-faba-490e-9680-d986a9635413-plugins-config-volume\") pod \"94fcd256-faba-490e-9680-d986a9635413\" (UID: \"94fcd256-faba-490e-9680-d986a9635413\") "
Oct 30 22:09:44.045606 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:44.045585    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66tv6\" (UniqueName: \"kubernetes.io/projected/72563fad-9b54-4005-a4eb-1466011220f8-kube-api-access-66tv6\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:44.045832 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:44.045808    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94fcd256-faba-490e-9680-d986a9635413-plugins-config-volume" pod "94fcd256-faba-490e-9680-d986a9635413" (UID: "94fcd256-faba-490e-9680-d986a9635413"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:44.047870 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:44.047849    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94fcd256-faba-490e-9680-d986a9635413-kube-api-access-chlq8" pod "94fcd256-faba-490e-9680-d986a9635413" (UID: "94fcd256-faba-490e-9680-d986a9635413"). InnerVolumeSpecName "kube-api-access-chlq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:44.151438 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:44.146408    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5801207-d12c-435e-b488-362e01e8c6d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5801207-d12c-435e-b488-362e01e8c6d4-maas-api-xks-ca\") pod \"b5801207-d12c-435e-b488-362e01e8c6d4\" (UID: \"b5801207-d12c-435e-b488-362e01e8c6d4\") "
Oct 30 22:09:44.151438 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:44.146476    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5801207-d12c-435e-b488-362e01e8c6d4-kube-api-access-bp7z9\" (UniqueName: \"kubernetes.io/projected/b5801207-d12c-435e-b488-362e01e8c6d4-kube-api-access-bp7z9\") pod \"b5801207-d12c-435e-b488-362e01e8c6d4\" (UID: \"b5801207-d12c-435e-b488-362e01e8c6d4\") "
Oct 30 22:09:44.151438 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:44.146536    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5801207-d12c-435e-b488-362e01e8c6d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5801207-d12c-435e-b488-362e01e8c6d4-maas-api-openshift-ca\") pod \"b5801207-d12c-435e-b488-362e01e8c6d4\" (UID: \"b5801207-d12c-435e-b488-362e01e8c6d4\") "
Oct 30 22:09:44.151438 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:44.147230    5087 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94fcd256-faba-490e-9680-d986a9635413-plugins-config-volume\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:44.151438 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:44.147254    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chlq8\" (UniqueName: \"kubernetes.io/projected/94fcd256-faba-490e-9680-d986a9635413-kube-api-access-chlq8\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:44.151438 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:44.147830    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5801207-d12c-435e-b488-362e01e8c6d4-maas-api-openshift-ca" pod "b5801207-d12c-435e-b488-362e01e8c6d4" (UID: "b5801207-d12c-435e-b488-362e01e8c6d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:44.151438 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:44.149894    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5801207-d12c-435e-b488-362e01e8c6d4-kube-api-access-bp7z9" pod "b5801207-d12c-435e-b488-362e01e8c6d4" (UID: "b5801207-d12c-435e-b488-362e01e8c6d4"). InnerVolumeSpecName "kube-api-access-bp7z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:44.151438 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:44.151086    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5801207-d12c-435e-b488-362e01e8c6d4-maas-api-xks-ca" pod "b5801207-d12c-435e-b488-362e01e8c6d4" (UID: "b5801207-d12c-435e-b488-362e01e8c6d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:44.248773 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:44.248728    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5801207-d12c-435e-b488-362e01e8c6d4-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:44.248773 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:44.248769    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp7z9\" (UniqueName: \"kubernetes.io/projected/b5801207-d12c-435e-b488-362e01e8c6d4-kube-api-access-bp7z9\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:44.249032 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:44.248785    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5801207-d12c-435e-b488-362e01e8c6d4-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:09:44.878544 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:44.878445    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-485d1e-58bccc79c9-jk9hv"
Oct 30 22:09:44.878544 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:44.878473    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-485d1e-58bccc79c9-jk9hv" event={"ID":"94fcd256-faba-490e-9680-d986a9635413","Type":"ContainerDied","Data":"81a2ce5cc2e7b89074d161c00625571e694104f608800c18055abac89c17c373"}
Oct 30 22:09:44.878544 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:44.878534    5087 scope.go:122] "RemoveContainer" containerID="b156f7d992db23f26da07c58f46ab9dcf72b827dff754a7b06614db837c7a684"
Oct 30 22:09:44.880208 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:44.880165    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c8c4d-mdhnv"
Oct 30 22:09:44.880440 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:44.880181    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c8c4d-mdhnv" event={"ID":"b5801207-d12c-435e-b488-362e01e8c6d4","Type":"ContainerDied","Data":"05f5863663c638ada514470e41a6abc2a688785e154117ed5dd1b36771e72f68"}
Oct 30 22:09:44.880545 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:44.880474    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05f5863663c638ada514470e41a6abc2a688785e154117ed5dd1b36771e72f68"
Oct 30 22:09:44.880545 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:44.880194    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-485d1e-66c964c857-fddzn"
Oct 30 22:09:45.043544 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:45.043505    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-485d1e-66c964c857-fddzn"]
Oct 30 22:09:45.047044 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:45.047018    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-485d1e-66c964c857-fddzn"]
Oct 30 22:09:45.205859 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:45.205773    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-485d1e-58bccc79c9-jk9hv"]
Oct 30 22:09:45.212202 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:45.212178    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-485d1e-58bccc79c9-jk9hv"]
Oct 30 22:09:45.710251 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:45.710219    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72563fad-9b54-4005-a4eb-1466011220f8" path="/var/lib/kubelet/pods/72563fad-9b54-4005-a4eb-1466011220f8/volumes"
Oct 30 22:09:45.710669 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:45.710655    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94fcd256-faba-490e-9680-d986a9635413" path="/var/lib/kubelet/pods/94fcd256-faba-490e-9680-d986a9635413/volumes"
Oct 30 22:09:47.169367 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:09:47.169313    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:09:57.984717 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:57.984684    5087 generic.go:356] "Generic (PLEG): container finished" podID="164e63e1-b309-4236-983a-2fd171af5b14" containerID="e1aeff8d7817ae5828f5734121dd9658c793153519f5374b39cc49105fd08bea" exitCode=0
Oct 30 22:09:57.985208 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:57.984746    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz" event={"ID":"164e63e1-b309-4236-983a-2fd171af5b14","Type":"ContainerDied","Data":"e1aeff8d7817ae5828f5734121dd9658c793153519f5374b39cc49105fd08bea"}
Oct 30 22:09:57.985208 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:57.985166    5087 scope.go:122] "RemoveContainer" containerID="e1aeff8d7817ae5828f5734121dd9658c793153519f5374b39cc49105fd08bea"
Oct 30 22:09:57.986749 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:57.986715    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:09:57.986808 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:57.986778    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:09:57.986844 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:57.986809    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:09:58.995657 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:58.995621    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz" event={"ID":"164e63e1-b309-4236-983a-2fd171af5b14","Type":"ContainerStarted","Data":"71bff2f05db10d65035ba080c86c05ff25df7ad6d47129088ed2b63d400c7f6d"}
Oct 30 22:09:59.210705 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:59.210668    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78cf83-8dltn"]
Oct 30 22:09:59.211330 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:59.211309    5087 cpu_manager.go:414] "containerMap: removing container" podUID="b5801207-d12c-435e-b488-362e01e8c6d4" containerName="revoke-keys"
Oct 30 22:09:59.211476 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:59.211332    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5801207-d12c-435e-b488-362e01e8c6d4" containerName="revoke-keys"
Oct 30 22:09:59.211476 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:59.211404    5087 cpu_manager.go:414] "containerMap: removing container" podUID="94fcd256-faba-490e-9680-d986a9635413" containerName="payload-pre-processing"
Oct 30 22:09:59.211476 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:59.211413    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94fcd256-faba-490e-9680-d986a9635413" containerName="payload-pre-processing"
Oct 30 22:09:59.211476 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:59.211456    5087 cpu_manager.go:414] "containerMap: removing container" podUID="72563fad-9b54-4005-a4eb-1466011220f8" containerName="payload-processing"
Oct 30 22:09:59.211476 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:59.211463    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72563fad-9b54-4005-a4eb-1466011220f8" containerName="payload-processing"
Oct 30 22:09:59.211730 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:59.211636    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5801207-d12c-435e-b488-362e01e8c6d4" containerName="revoke-keys"
Oct 30 22:09:59.211730 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:59.211656    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="94fcd256-faba-490e-9680-d986a9635413" containerName="payload-pre-processing"
Oct 30 22:09:59.211730 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:59.211674    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="72563fad-9b54-4005-a4eb-1466011220f8" containerName="payload-processing"
Oct 30 22:09:59.214434 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:59.214412    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78cf83-8dltn"
Oct 30 22:09:59.270005 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:59.269783    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:09:59.270005 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:59.269861    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:09:59.273807 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:59.273768    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78cf83-8dltn"]
Oct 30 22:09:59.295247 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:59.295214    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-78cf83-8dltn\" (UID: \"2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78cf83-8dltn"
Oct 30 22:09:59.295444 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:59.295333    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xwvw\" (UniqueName: \"kubernetes.io/projected/2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7-kube-api-access-4xwvw\") pod \"maas-api-revoke-keys-e2e-api-a-78cf83-8dltn\" (UID: \"2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78cf83-8dltn"
Oct 30 22:09:59.295536 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:59.295494    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-78cf83-8dltn\" (UID: \"2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78cf83-8dltn"
Oct 30 22:09:59.397288 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:59.397249    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-78cf83-8dltn\" (UID: \"2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78cf83-8dltn"
Oct 30 22:09:59.397500 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:59.397440    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-78cf83-8dltn\" (UID: \"2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78cf83-8dltn"
Oct 30 22:09:59.397500 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:59.397479    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xwvw\" (UniqueName: \"kubernetes.io/projected/2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7-kube-api-access-4xwvw\") pod \"maas-api-revoke-keys-e2e-api-a-78cf83-8dltn\" (UID: \"2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78cf83-8dltn"
Oct 30 22:09:59.398108 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:59.398079    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-78cf83-8dltn\" (UID: \"2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78cf83-8dltn"
Oct 30 22:09:59.399817 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:59.399799    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-78cf83-8dltn\" (UID: \"2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78cf83-8dltn"
Oct 30 22:09:59.467378 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:59.458024    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xwvw\" (UniqueName: \"kubernetes.io/projected/2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7-kube-api-access-4xwvw\") pod \"maas-api-revoke-keys-e2e-api-a-78cf83-8dltn\" (UID: \"2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78cf83-8dltn"
Oct 30 22:09:59.528148 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:59.528069    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78cf83-8dltn"
Oct 30 22:09:59.803079 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:59.803000    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:09:59.803638 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:59.803611    5087 patch_prober.go:29] interesting pod/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 22:09:59.803727 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:59.803671    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz" podUID="164e63e1-b309-4236-983a-2fd171af5b14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 22:09:59.972604 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:09:59.972571    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78cf83-8dltn"]
Oct 30 22:09:59.977158 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:09:59.977129    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc8a1ab_daf0_4fcb_b1f3_6dc730f1a0f7.slice/crio-b0ae6b35b4f830c90ce2c5973eb3e995dd60b2c283de3003c8ae5684ab88f594 WatchSource:0}: Error finding container b0ae6b35b4f830c90ce2c5973eb3e995dd60b2c283de3003c8ae5684ab88f594: Status 404 returned error can't find the container with id b0ae6b35b4f830c90ce2c5973eb3e995dd60b2c283de3003c8ae5684ab88f594
Oct 30 22:10:00.004377 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:00.004325    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78cf83-8dltn" event={"ID":"2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7","Type":"ContainerStarted","Data":"b0ae6b35b4f830c90ce2c5973eb3e995dd60b2c283de3003c8ae5684ab88f594"}
Oct 30 22:10:00.802234 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:00.802197    5087 patch_prober.go:29] interesting pod/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 22:10:00.802459 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:00.802247    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz" podUID="164e63e1-b309-4236-983a-2fd171af5b14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 22:10:01.012785 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:01.012746    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78cf83-8dltn" event={"ID":"2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7","Type":"ContainerStarted","Data":"3dff0e4cbdba5f4ad27ebb3677cb7d5fadc58147d7cf0b3bc5c0005ac9e98dff"}
Oct 30 22:10:01.134374 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:01.134309    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78cf83-8dltn" podStartSLOduration=2.134294299 podStartE2EDuration="2.134294299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:01.131698587 +0000 UTC m=+2195.606202565" watchObservedRunningTime="2026-10-30 22:10:01.134294299 +0000 UTC m=+2195.608798271"
Oct 30 22:10:01.803086 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:01.803047    5087 patch_prober.go:29] interesting pod/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 22:10:01.803276 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:01.803105    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz" podUID="164e63e1-b309-4236-983a-2fd171af5b14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 22:10:02.021015 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:02.020981    5087 generic.go:356] "Generic (PLEG): container finished" podID="2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7" containerID="3dff0e4cbdba5f4ad27ebb3677cb7d5fadc58147d7cf0b3bc5c0005ac9e98dff" exitCode=0
Oct 30 22:10:02.021426 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:02.021067    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78cf83-8dltn" event={"ID":"2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7","Type":"ContainerDied","Data":"3dff0e4cbdba5f4ad27ebb3677cb7d5fadc58147d7cf0b3bc5c0005ac9e98dff"}
Oct 30 22:10:02.802440 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:02.802407    5087 patch_prober.go:29] interesting pod/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 22:10:02.802653 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:02.802463    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz" podUID="164e63e1-b309-4236-983a-2fd171af5b14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 22:10:03.283095 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:03.283068    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78cf83-8dltn"
Oct 30 22:10:03.380964 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:03.380875    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7-kube-api-access-4xwvw\" (UniqueName: \"kubernetes.io/projected/2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7-kube-api-access-4xwvw\") pod \"2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7\" (UID: \"2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7\") "
Oct 30 22:10:03.381152 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:03.381042    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7-maas-api-openshift-ca\") pod \"2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7\" (UID: \"2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7\") "
Oct 30 22:10:03.381152 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:03.381131    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7-maas-api-xks-ca\") pod \"2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7\" (UID: \"2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7\") "
Oct 30 22:10:03.381481 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:03.381454    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7-maas-api-openshift-ca" pod "2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7" (UID: "2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:03.381662 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:03.381644    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:03.383466 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:03.383438    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7-kube-api-access-4xwvw" pod "2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7" (UID: "2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7"). InnerVolumeSpecName "kube-api-access-4xwvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:03.384552 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:03.383854    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7-maas-api-xks-ca" pod "2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7" (UID: "2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:03.482480 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:03.482430    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xwvw\" (UniqueName: \"kubernetes.io/projected/2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7-kube-api-access-4xwvw\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:03.482480 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:03.482477    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:03.802972 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:03.802936    5087 patch_prober.go:29] interesting pod/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 30 22:10:03.803125 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:03.802988    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz" podUID="164e63e1-b309-4236-983a-2fd171af5b14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 30 22:10:04.036306 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:04.036267    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78cf83-8dltn" event={"ID":"2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7","Type":"ContainerDied","Data":"b0ae6b35b4f830c90ce2c5973eb3e995dd60b2c283de3003c8ae5684ab88f594"}
Oct 30 22:10:04.036502 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:04.036321    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78cf83-8dltn"
Oct 30 22:10:04.036502 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:04.036388    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0ae6b35b4f830c90ce2c5973eb3e995dd60b2c283de3003c8ae5684ab88f594"
Oct 30 22:10:04.807171 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:04.807144    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:10:04.807660 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:04.807637    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:10:04.808609 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:04.808584    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:10:07.477411 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:07.477378    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"]
Oct 30 22:10:07.477872 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:07.477556    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w" podUID="faa7e534-c66d-4670-83cd-c170bcc355d6" containerName="istio-proxy" containerID="cri-o://a913b4148c77ed1adce59741beac8509d17d1e9fba2075b195ecdd13ff0b4933" gracePeriod=30
Oct 30 22:10:07.833058 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:07.833019    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29"]
Oct 30 22:10:07.833322 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:07.833308    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-ea3cb2-5jw29"]
Oct 30 22:10:08.244827 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:08.244789    5087 patch_prober.go:29] interesting pod/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:10:08.244998 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:08.244863    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w" podUID="faa7e534-c66d-4670-83cd-c170bcc355d6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:10:09.691517 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:09.691482    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-78cf83-796db64fff-7ph2p"]
Oct 30 22:10:09.709616 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:09.709582    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67c150f1-88ba-4010-b3ea-750b1cc9db58" path="/var/lib/kubelet/pods/67c150f1-88ba-4010-b3ea-750b1cc9db58/volumes"
Oct 30 22:10:12.098449 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.098415    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb"]
Oct 30 22:10:12.098954 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.098618    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb" podUID="6b3238c7-a167-49c6-8d68-6c7af7145853" containerName="maas-api" containerID="cri-o://b5187f68ca27e0bc6958daaef60d78261548a1bfd19772c5833cb5b454bd443c" gracePeriod=30
Oct 30 22:10:12.439671 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.439636    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb"
Oct 30 22:10:12.514632 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.514602    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b3238c7-a167-49c6-8d68-6c7af7145853-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3238c7-a167-49c6-8d68-6c7af7145853-openshift-service-ca\") pod \"6b3238c7-a167-49c6-8d68-6c7af7145853\" (UID: \"6b3238c7-a167-49c6-8d68-6c7af7145853\") "
Oct 30 22:10:12.514806 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.514730    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b3238c7-a167-49c6-8d68-6c7af7145853-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b3238c7-a167-49c6-8d68-6c7af7145853-metrics-certs\") pod \"6b3238c7-a167-49c6-8d68-6c7af7145853\" (UID: \"6b3238c7-a167-49c6-8d68-6c7af7145853\") "
Oct 30 22:10:12.514806 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.514791    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b3238c7-a167-49c6-8d68-6c7af7145853-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b3238c7-a167-49c6-8d68-6c7af7145853-maas-api-tls\") pod \"6b3238c7-a167-49c6-8d68-6c7af7145853\" (UID: \"6b3238c7-a167-49c6-8d68-6c7af7145853\") "
Oct 30 22:10:12.514923 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.514823    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b3238c7-a167-49c6-8d68-6c7af7145853-kube-api-access-zfthx\" (UniqueName: \"kubernetes.io/projected/6b3238c7-a167-49c6-8d68-6c7af7145853-kube-api-access-zfthx\") pod \"6b3238c7-a167-49c6-8d68-6c7af7145853\" (UID: \"6b3238c7-a167-49c6-8d68-6c7af7145853\") "
Oct 30 22:10:12.515961 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.515935    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b3238c7-a167-49c6-8d68-6c7af7145853-openshift-service-ca" pod "6b3238c7-a167-49c6-8d68-6c7af7145853" (UID: "6b3238c7-a167-49c6-8d68-6c7af7145853"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:12.517928 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.517889    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b3238c7-a167-49c6-8d68-6c7af7145853-metrics-certs" pod "6b3238c7-a167-49c6-8d68-6c7af7145853" (UID: "6b3238c7-a167-49c6-8d68-6c7af7145853"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:12.518148 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.518124    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3238c7-a167-49c6-8d68-6c7af7145853-kube-api-access-zfthx" pod "6b3238c7-a167-49c6-8d68-6c7af7145853" (UID: "6b3238c7-a167-49c6-8d68-6c7af7145853"). InnerVolumeSpecName "kube-api-access-zfthx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:12.518496 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.518476    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b3238c7-a167-49c6-8d68-6c7af7145853-maas-api-tls" pod "6b3238c7-a167-49c6-8d68-6c7af7145853" (UID: "6b3238c7-a167-49c6-8d68-6c7af7145853"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:12.621007 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.620570    5087 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b3238c7-a167-49c6-8d68-6c7af7145853-openshift-service-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:12.621007 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.620612    5087 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b3238c7-a167-49c6-8d68-6c7af7145853-metrics-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:12.621007 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.620628    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b3238c7-a167-49c6-8d68-6c7af7145853-maas-api-tls\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:12.621007 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.620642    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfthx\" (UniqueName: \"kubernetes.io/projected/6b3238c7-a167-49c6-8d68-6c7af7145853-kube-api-access-zfthx\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:12.810845 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.810819    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 22:10:12.822796 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.822772    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-credential-socket\") pod \"faa7e534-c66d-4670-83cd-c170bcc355d6\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") "
Oct 30 22:10:12.822919 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.822830    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa7e534-c66d-4670-83cd-c170bcc355d6-istio-token\" (UniqueName: \"kubernetes.io/projected/faa7e534-c66d-4670-83cd-c170bcc355d6-istio-token\") pod \"faa7e534-c66d-4670-83cd-c170bcc355d6\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") "
Oct 30 22:10:12.822919 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.822860    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-istio-envoy\") pod \"faa7e534-c66d-4670-83cd-c170bcc355d6\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") "
Oct 30 22:10:12.822919 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.822896    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faa7e534-c66d-4670-83cd-c170bcc355d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa7e534-c66d-4670-83cd-c170bcc355d6-istiod-ca-cert\") pod \"faa7e534-c66d-4670-83cd-c170bcc355d6\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") "
Oct 30 22:10:12.823056 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.822947    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/faa7e534-c66d-4670-83cd-c170bcc355d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa7e534-c66d-4670-83cd-c170bcc355d6-istio-podinfo\") pod \"faa7e534-c66d-4670-83cd-c170bcc355d6\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") "
Oct 30 22:10:12.823056 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.822982    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-workload-certs\") pod \"faa7e534-c66d-4670-83cd-c170bcc355d6\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") "
Oct 30 22:10:12.823056 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.823021    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-credential-socket" pod "faa7e534-c66d-4670-83cd-c170bcc355d6" (UID: "faa7e534-c66d-4670-83cd-c170bcc355d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:12.823213 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.823088    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-workload-socket\") pod \"faa7e534-c66d-4670-83cd-c170bcc355d6\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") "
Oct 30 22:10:12.823213 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.823172    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faa7e534-c66d-4670-83cd-c170bcc355d6-kube-api-access-z2tf8\" (UniqueName: \"kubernetes.io/projected/faa7e534-c66d-4670-83cd-c170bcc355d6-kube-api-access-z2tf8\") pod \"faa7e534-c66d-4670-83cd-c170bcc355d6\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") "
Oct 30 22:10:12.823334 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.823226    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-istio-data\") pod \"faa7e534-c66d-4670-83cd-c170bcc355d6\" (UID: \"faa7e534-c66d-4670-83cd-c170bcc355d6\") "
Oct 30 22:10:12.823334 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.823251    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-workload-certs" pod "faa7e534-c66d-4670-83cd-c170bcc355d6" (UID: "faa7e534-c66d-4670-83cd-c170bcc355d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:12.823506 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.823389    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faa7e534-c66d-4670-83cd-c170bcc355d6-istiod-ca-cert" pod "faa7e534-c66d-4670-83cd-c170bcc355d6" (UID: "faa7e534-c66d-4670-83cd-c170bcc355d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:12.823598 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.823576    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-istio-data" pod "faa7e534-c66d-4670-83cd-c170bcc355d6" (UID: "faa7e534-c66d-4670-83cd-c170bcc355d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:12.823929 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.823890    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-workload-socket" pod "faa7e534-c66d-4670-83cd-c170bcc355d6" (UID: "faa7e534-c66d-4670-83cd-c170bcc355d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:12.824067 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.824048    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/faa7e534-c66d-4670-83cd-c170bcc355d6-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:12.824135 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.824072    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:12.824135 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.824087    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:12.824135 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.824101    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:12.824135 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.824116    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:12.825253 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.825233    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa7e534-c66d-4670-83cd-c170bcc355d6-istio-token" pod "faa7e534-c66d-4670-83cd-c170bcc355d6" (UID: "faa7e534-c66d-4670-83cd-c170bcc355d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:12.825326 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.825311    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/faa7e534-c66d-4670-83cd-c170bcc355d6-istio-podinfo" pod "faa7e534-c66d-4670-83cd-c170bcc355d6" (UID: "faa7e534-c66d-4670-83cd-c170bcc355d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:12.825745 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.825729    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-istio-envoy" pod "faa7e534-c66d-4670-83cd-c170bcc355d6" (UID: "faa7e534-c66d-4670-83cd-c170bcc355d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:12.825965 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.825947    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faa7e534-c66d-4670-83cd-c170bcc355d6-kube-api-access-z2tf8" pod "faa7e534-c66d-4670-83cd-c170bcc355d6" (UID: "faa7e534-c66d-4670-83cd-c170bcc355d6"). InnerVolumeSpecName "kube-api-access-z2tf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:12.925038 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.925005    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/faa7e534-c66d-4670-83cd-c170bcc355d6-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:12.925038 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.925034    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2tf8\" (UniqueName: \"kubernetes.io/projected/faa7e534-c66d-4670-83cd-c170bcc355d6-kube-api-access-z2tf8\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:12.925038 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.925046    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/faa7e534-c66d-4670-83cd-c170bcc355d6-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:12.925289 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:12.925055    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/faa7e534-c66d-4670-83cd-c170bcc355d6-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:13.126188 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:13.126159    5087 generic.go:356] "Generic (PLEG): container finished" podID="6b3238c7-a167-49c6-8d68-6c7af7145853" containerID="b5187f68ca27e0bc6958daaef60d78261548a1bfd19772c5833cb5b454bd443c" exitCode=0
Oct 30 22:10:13.126655 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:13.126208    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb"
Oct 30 22:10:13.126655 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:13.126252    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb" event={"ID":"6b3238c7-a167-49c6-8d68-6c7af7145853","Type":"ContainerDied","Data":"b5187f68ca27e0bc6958daaef60d78261548a1bfd19772c5833cb5b454bd443c"}
Oct 30 22:10:13.126655 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:13.126306    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb" event={"ID":"6b3238c7-a167-49c6-8d68-6c7af7145853","Type":"ContainerDied","Data":"519e5ea31203c8f36db37dfbdf9a06ef8f5300a11344723ed3e785086e5b71e1"}
Oct 30 22:10:13.126655 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:13.126334    5087 scope.go:122] "RemoveContainer" containerID="b5187f68ca27e0bc6958daaef60d78261548a1bfd19772c5833cb5b454bd443c"
Oct 30 22:10:13.128082 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:13.128063    5087 generic.go:356] "Generic (PLEG): container finished" podID="faa7e534-c66d-4670-83cd-c170bcc355d6" containerID="a913b4148c77ed1adce59741beac8509d17d1e9fba2075b195ecdd13ff0b4933" exitCode=0
Oct 30 22:10:13.128178 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:13.128124    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"
Oct 30 22:10:13.128178 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:13.128137    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w" event={"ID":"faa7e534-c66d-4670-83cd-c170bcc355d6","Type":"ContainerDied","Data":"a913b4148c77ed1adce59741beac8509d17d1e9fba2075b195ecdd13ff0b4933"}
Oct 30 22:10:13.128299 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:13.128180    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w" event={"ID":"faa7e534-c66d-4670-83cd-c170bcc355d6","Type":"ContainerDied","Data":"2712e51e68b7262d214a79f8a5cb3c4419930de1c651de2502aa0b0d031cf80c"}
Oct 30 22:10:13.140159 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:13.140141    5087 scope.go:122] "RemoveContainer" containerID="b5187f68ca27e0bc6958daaef60d78261548a1bfd19772c5833cb5b454bd443c"
Oct 30 22:10:13.140501 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:10:13.140483    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5187f68ca27e0bc6958daaef60d78261548a1bfd19772c5833cb5b454bd443c\": container with ID starting with b5187f68ca27e0bc6958daaef60d78261548a1bfd19772c5833cb5b454bd443c not found: ID does not exist" containerID="b5187f68ca27e0bc6958daaef60d78261548a1bfd19772c5833cb5b454bd443c"
Oct 30 22:10:13.140588 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:13.140513    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5187f68ca27e0bc6958daaef60d78261548a1bfd19772c5833cb5b454bd443c"} err="failed to get container status \"b5187f68ca27e0bc6958daaef60d78261548a1bfd19772c5833cb5b454bd443c\": rpc error: code = NotFound desc = could not find container \"b5187f68ca27e0bc6958daaef60d78261548a1bfd19772c5833cb5b454bd443c\": container with ID starting with b5187f68ca27e0bc6958daaef60d78261548a1bfd19772c5833cb5b454bd443c not found: ID does not exist"
Oct 30 22:10:13.140588 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:13.140538    5087 scope.go:122] "RemoveContainer" containerID="a913b4148c77ed1adce59741beac8509d17d1e9fba2075b195ecdd13ff0b4933"
Oct 30 22:10:13.150577 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:13.150561    5087 scope.go:122] "RemoveContainer" containerID="a913b4148c77ed1adce59741beac8509d17d1e9fba2075b195ecdd13ff0b4933"
Oct 30 22:10:13.150828 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:10:13.150809    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a913b4148c77ed1adce59741beac8509d17d1e9fba2075b195ecdd13ff0b4933\": container with ID starting with a913b4148c77ed1adce59741beac8509d17d1e9fba2075b195ecdd13ff0b4933 not found: ID does not exist" containerID="a913b4148c77ed1adce59741beac8509d17d1e9fba2075b195ecdd13ff0b4933"
Oct 30 22:10:13.150893 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:13.150835    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a913b4148c77ed1adce59741beac8509d17d1e9fba2075b195ecdd13ff0b4933"} err="failed to get container status \"a913b4148c77ed1adce59741beac8509d17d1e9fba2075b195ecdd13ff0b4933\": rpc error: code = NotFound desc = could not find container \"a913b4148c77ed1adce59741beac8509d17d1e9fba2075b195ecdd13ff0b4933\": container with ID starting with a913b4148c77ed1adce59741beac8509d17d1e9fba2075b195ecdd13ff0b4933 not found: ID does not exist"
Oct 30 22:10:13.290966 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:13.290935    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"]
Oct 30 22:10:13.296576 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:13.296547    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-tzj7w"]
Oct 30 22:10:13.424774 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:13.424678    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb"]
Oct 30 22:10:13.439742 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:13.439713    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-78cf83-6bfbb6f6b6-84qxb"]
Oct 30 22:10:13.714581 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:13.714505    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b3238c7-a167-49c6-8d68-6c7af7145853" path="/var/lib/kubelet/pods/6b3238c7-a167-49c6-8d68-6c7af7145853/volumes"
Oct 30 22:10:13.715223 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:13.715203    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faa7e534-c66d-4670-83cd-c170bcc355d6" path="/var/lib/kubelet/pods/faa7e534-c66d-4670-83cd-c170bcc355d6/volumes"
Oct 30 22:10:14.434399 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:14.426656    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78cf83-8dltn"]
Oct 30 22:10:14.465695 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:14.465657    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-78cf83-8dltn"]
Oct 30 22:10:14.695020 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:14.694914    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78cf83-796db64fff-7ph2p" podUID="ab4753ec-050c-4e05-b558-cc77998d92a6" containerName="payload-pre-processing" containerID="cri-o://c3e6f05d371590674117d3cee45591d9d3e1d997f263da7819d32b3e0a9f1197" gracePeriod=25
Oct 30 22:10:14.966277 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:14.966254    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78cf83-796db64fff-7ph2p"
Oct 30 22:10:15.054650 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:15.054622    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab4753ec-050c-4e05-b558-cc77998d92a6-kube-api-access-644z7\" (UniqueName: \"kubernetes.io/projected/ab4753ec-050c-4e05-b558-cc77998d92a6-kube-api-access-644z7\") pod \"ab4753ec-050c-4e05-b558-cc77998d92a6\" (UID: \"ab4753ec-050c-4e05-b558-cc77998d92a6\") "
Oct 30 22:10:15.054811 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:15.054693    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab4753ec-050c-4e05-b558-cc77998d92a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab4753ec-050c-4e05-b558-cc77998d92a6-plugins-config-volume\") pod \"ab4753ec-050c-4e05-b558-cc77998d92a6\" (UID: \"ab4753ec-050c-4e05-b558-cc77998d92a6\") "
Oct 30 22:10:15.055085 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:15.055053    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab4753ec-050c-4e05-b558-cc77998d92a6-plugins-config-volume" pod "ab4753ec-050c-4e05-b558-cc77998d92a6" (UID: "ab4753ec-050c-4e05-b558-cc77998d92a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:15.056932 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:15.056914    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab4753ec-050c-4e05-b558-cc77998d92a6-kube-api-access-644z7" pod "ab4753ec-050c-4e05-b558-cc77998d92a6" (UID: "ab4753ec-050c-4e05-b558-cc77998d92a6"). InnerVolumeSpecName "kube-api-access-644z7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:15.146328 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:15.146279    5087 generic.go:356] "Generic (PLEG): container finished" podID="ab4753ec-050c-4e05-b558-cc77998d92a6" containerID="c3e6f05d371590674117d3cee45591d9d3e1d997f263da7819d32b3e0a9f1197" exitCode=0
Oct 30 22:10:15.146507 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:15.146438    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78cf83-796db64fff-7ph2p"
Oct 30 22:10:15.146507 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:15.146473    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78cf83-796db64fff-7ph2p" event={"ID":"ab4753ec-050c-4e05-b558-cc77998d92a6","Type":"ContainerDied","Data":"c3e6f05d371590674117d3cee45591d9d3e1d997f263da7819d32b3e0a9f1197"}
Oct 30 22:10:15.146640 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:15.146545    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-78cf83-796db64fff-7ph2p" event={"ID":"ab4753ec-050c-4e05-b558-cc77998d92a6","Type":"ContainerDied","Data":"d9c14a89be6f26e6981e9d47d8ceb3749793fbb6474af601cc6e42361340bcfc"}
Oct 30 22:10:15.146640 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:15.146567    5087 scope.go:122] "RemoveContainer" containerID="c3e6f05d371590674117d3cee45591d9d3e1d997f263da7819d32b3e0a9f1197"
Oct 30 22:10:15.155956 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:15.155931    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-644z7\" (UniqueName: \"kubernetes.io/projected/ab4753ec-050c-4e05-b558-cc77998d92a6-kube-api-access-644z7\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.155956 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:15.155957    5087 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab4753ec-050c-4e05-b558-cc77998d92a6-plugins-config-volume\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:15.158872 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:15.158850    5087 scope.go:122] "RemoveContainer" containerID="c3e6f05d371590674117d3cee45591d9d3e1d997f263da7819d32b3e0a9f1197"
Oct 30 22:10:15.159148 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:10:15.159128    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3e6f05d371590674117d3cee45591d9d3e1d997f263da7819d32b3e0a9f1197\": container with ID starting with c3e6f05d371590674117d3cee45591d9d3e1d997f263da7819d32b3e0a9f1197 not found: ID does not exist" containerID="c3e6f05d371590674117d3cee45591d9d3e1d997f263da7819d32b3e0a9f1197"
Oct 30 22:10:15.159194 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:15.159156    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3e6f05d371590674117d3cee45591d9d3e1d997f263da7819d32b3e0a9f1197"} err="failed to get container status \"c3e6f05d371590674117d3cee45591d9d3e1d997f263da7819d32b3e0a9f1197\": rpc error: code = NotFound desc = could not find container \"c3e6f05d371590674117d3cee45591d9d3e1d997f263da7819d32b3e0a9f1197\": container with ID starting with c3e6f05d371590674117d3cee45591d9d3e1d997f263da7819d32b3e0a9f1197 not found: ID does not exist"
Oct 30 22:10:15.310198 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:15.310146    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-78cf83-796db64fff-7ph2p"]
Oct 30 22:10:15.315410 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:15.315375    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-78cf83-796db64fff-7ph2p"]
Oct 30 22:10:15.710553 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:15.710464    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7" path="/var/lib/kubelet/pods/2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7/volumes"
Oct 30 22:10:15.711094 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:15.711056    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab4753ec-050c-4e05-b558-cc77998d92a6" path="/var/lib/kubelet/pods/ab4753ec-050c-4e05-b558-cc77998d92a6/volumes"
Oct 30 22:10:15.712800 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:15.712777    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9d4043-927m5"]
Oct 30 22:10:15.715600 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:15.715583    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-9d4043-927m5"]
Oct 30 22:10:16.240372 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:16.240324    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6cd200-gzbct"]
Oct 30 22:10:16.240963 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:16.240942    5087 cpu_manager.go:414] "containerMap: removing container" podUID="faa7e534-c66d-4670-83cd-c170bcc355d6" containerName="istio-proxy"
Oct 30 22:10:16.240963 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:16.240959    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faa7e534-c66d-4670-83cd-c170bcc355d6" containerName="istio-proxy"
Oct 30 22:10:16.241177 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:16.240971    5087 cpu_manager.go:414] "containerMap: removing container" podUID="ab4753ec-050c-4e05-b558-cc77998d92a6" containerName="payload-pre-processing"
Oct 30 22:10:16.241177 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:16.240979    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab4753ec-050c-4e05-b558-cc77998d92a6" containerName="payload-pre-processing"
Oct 30 22:10:16.241177 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:16.241002    5087 cpu_manager.go:414] "containerMap: removing container" podUID="6b3238c7-a167-49c6-8d68-6c7af7145853" containerName="maas-api"
Oct 30 22:10:16.241177 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:16.241007    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b3238c7-a167-49c6-8d68-6c7af7145853" containerName="maas-api"
Oct 30 22:10:16.241177 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:16.241030    5087 cpu_manager.go:414] "containerMap: removing container" podUID="2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7" containerName="revoke-keys"
Oct 30 22:10:16.241177 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:16.241034    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7" containerName="revoke-keys"
Oct 30 22:10:16.241177 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:16.241118    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="faa7e534-c66d-4670-83cd-c170bcc355d6" containerName="istio-proxy"
Oct 30 22:10:16.241177 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:16.241129    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab4753ec-050c-4e05-b558-cc77998d92a6" containerName="payload-pre-processing"
Oct 30 22:10:16.241177 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:16.241138    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b3238c7-a167-49c6-8d68-6c7af7145853" containerName="maas-api"
Oct 30 22:10:16.241177 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:16.241147    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bc8a1ab-daf0-4fcb-b1f3-6dc730f1a0f7" containerName="revoke-keys"
Oct 30 22:10:16.245936 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:16.245917    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6cd200-gzbct"
Oct 30 22:10:16.269413 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:16.269368    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/493e39ae-056c-4407-be7c-9e2f4b4ac812-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6cd200-gzbct\" (UID: \"493e39ae-056c-4407-be7c-9e2f4b4ac812\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6cd200-gzbct"
Oct 30 22:10:16.269536 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:16.269472    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7srnd\" (UniqueName: \"kubernetes.io/projected/493e39ae-056c-4407-be7c-9e2f4b4ac812-kube-api-access-7srnd\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6cd200-gzbct\" (UID: \"493e39ae-056c-4407-be7c-9e2f4b4ac812\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6cd200-gzbct"
Oct 30 22:10:16.269536 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:16.269530    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/493e39ae-056c-4407-be7c-9e2f4b4ac812-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6cd200-gzbct\" (UID: \"493e39ae-056c-4407-be7c-9e2f4b4ac812\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6cd200-gzbct"
Oct 30 22:10:16.297900 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:16.297871    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:10:16.298018 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:16.297973    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:10:16.351122 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:16.351090    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6cd200-gzbct"]
Oct 30 22:10:16.370285 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:16.370256    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/493e39ae-056c-4407-be7c-9e2f4b4ac812-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6cd200-gzbct\" (UID: \"493e39ae-056c-4407-be7c-9e2f4b4ac812\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6cd200-gzbct"
Oct 30 22:10:16.370446 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:16.370299    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7srnd\" (UniqueName: \"kubernetes.io/projected/493e39ae-056c-4407-be7c-9e2f4b4ac812-kube-api-access-7srnd\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6cd200-gzbct\" (UID: \"493e39ae-056c-4407-be7c-9e2f4b4ac812\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6cd200-gzbct"
Oct 30 22:10:16.370446 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:16.370362    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/493e39ae-056c-4407-be7c-9e2f4b4ac812-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6cd200-gzbct\" (UID: \"493e39ae-056c-4407-be7c-9e2f4b4ac812\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6cd200-gzbct"
Oct 30 22:10:16.370959 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:16.370936    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/493e39ae-056c-4407-be7c-9e2f4b4ac812-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6cd200-gzbct\" (UID: \"493e39ae-056c-4407-be7c-9e2f4b4ac812\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6cd200-gzbct"
Oct 30 22:10:16.372905 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:16.372883    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/493e39ae-056c-4407-be7c-9e2f4b4ac812-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6cd200-gzbct\" (UID: \"493e39ae-056c-4407-be7c-9e2f4b4ac812\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6cd200-gzbct"
Oct 30 22:10:16.427741 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:16.427706    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7srnd\" (UniqueName: \"kubernetes.io/projected/493e39ae-056c-4407-be7c-9e2f4b4ac812-kube-api-access-7srnd\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-6cd200-gzbct\" (UID: \"493e39ae-056c-4407-be7c-9e2f4b4ac812\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6cd200-gzbct"
Oct 30 22:10:16.560029 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:16.559939    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6cd200-gzbct"
Oct 30 22:10:16.942814 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:16.942786    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6cd200-gzbct"]
Oct 30 22:10:16.944579 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:10:16.944545    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod493e39ae_056c_4407_be7c_9e2f4b4ac812.slice/crio-5e87cf567f0b711ef607ebe6e72cf58ec093e130fcfafab8a6b1097a28deaa05 WatchSource:0}: Error finding container 5e87cf567f0b711ef607ebe6e72cf58ec093e130fcfafab8a6b1097a28deaa05: Status 404 returned error can't find the container with id 5e87cf567f0b711ef607ebe6e72cf58ec093e130fcfafab8a6b1097a28deaa05
Oct 30 22:10:17.163902 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:17.163870    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6cd200-gzbct" event={"ID":"493e39ae-056c-4407-be7c-9e2f4b4ac812","Type":"ContainerStarted","Data":"5e87cf567f0b711ef607ebe6e72cf58ec093e130fcfafab8a6b1097a28deaa05"}
Oct 30 22:10:17.176926 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:10:17.176894    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:10:17.710581 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:17.710489    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ba89675-526d-4f5d-b8ff-d676460179fd" path="/var/lib/kubelet/pods/3ba89675-526d-4f5d-b8ff-d676460179fd/volumes"
Oct 30 22:10:18.172778 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:18.172745    5087 generic.go:356] "Generic (PLEG): container finished" podID="493e39ae-056c-4407-be7c-9e2f4b4ac812" containerID="1ec58bd1acd788ba0bc9998c85c6e072aa1a20b46744e9f44b97c3bfc7c0a727" exitCode=0
Oct 30 22:10:18.173204 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:18.172836    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6cd200-gzbct" event={"ID":"493e39ae-056c-4407-be7c-9e2f4b4ac812","Type":"ContainerDied","Data":"1ec58bd1acd788ba0bc9998c85c6e072aa1a20b46744e9f44b97c3bfc7c0a727"}
Oct 30 22:10:19.335782 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:19.335755    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6cd200-gzbct"
Oct 30 22:10:19.410179 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:19.410141    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/493e39ae-056c-4407-be7c-9e2f4b4ac812-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/493e39ae-056c-4407-be7c-9e2f4b4ac812-maas-api-openshift-ca\") pod \"493e39ae-056c-4407-be7c-9e2f4b4ac812\" (UID: \"493e39ae-056c-4407-be7c-9e2f4b4ac812\") "
Oct 30 22:10:19.410380 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:19.410203    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/493e39ae-056c-4407-be7c-9e2f4b4ac812-kube-api-access-7srnd\" (UniqueName: \"kubernetes.io/projected/493e39ae-056c-4407-be7c-9e2f4b4ac812-kube-api-access-7srnd\") pod \"493e39ae-056c-4407-be7c-9e2f4b4ac812\" (UID: \"493e39ae-056c-4407-be7c-9e2f4b4ac812\") "
Oct 30 22:10:19.410380 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:19.410317    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/493e39ae-056c-4407-be7c-9e2f4b4ac812-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/493e39ae-056c-4407-be7c-9e2f4b4ac812-maas-api-xks-ca\") pod \"493e39ae-056c-4407-be7c-9e2f4b4ac812\" (UID: \"493e39ae-056c-4407-be7c-9e2f4b4ac812\") "
Oct 30 22:10:19.410575 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:19.410551    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/493e39ae-056c-4407-be7c-9e2f4b4ac812-maas-api-openshift-ca" pod "493e39ae-056c-4407-be7c-9e2f4b4ac812" (UID: "493e39ae-056c-4407-be7c-9e2f4b4ac812"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:19.410823 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:19.410800    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/493e39ae-056c-4407-be7c-9e2f4b4ac812-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:19.412697 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:19.412671    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/493e39ae-056c-4407-be7c-9e2f4b4ac812-maas-api-xks-ca" pod "493e39ae-056c-4407-be7c-9e2f4b4ac812" (UID: "493e39ae-056c-4407-be7c-9e2f4b4ac812"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:19.412697 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:19.412678    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/493e39ae-056c-4407-be7c-9e2f4b4ac812-kube-api-access-7srnd" pod "493e39ae-056c-4407-be7c-9e2f4b4ac812" (UID: "493e39ae-056c-4407-be7c-9e2f4b4ac812"). InnerVolumeSpecName "kube-api-access-7srnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:19.512441 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:19.512408    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7srnd\" (UniqueName: \"kubernetes.io/projected/493e39ae-056c-4407-be7c-9e2f4b4ac812-kube-api-access-7srnd\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:19.512441 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:19.512436    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/493e39ae-056c-4407-be7c-9e2f4b4ac812-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:20.188420 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:20.188388    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6cd200-gzbct"
Oct 30 22:10:20.188584 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:20.188393    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6cd200-gzbct" event={"ID":"493e39ae-056c-4407-be7c-9e2f4b4ac812","Type":"ContainerDied","Data":"5e87cf567f0b711ef607ebe6e72cf58ec093e130fcfafab8a6b1097a28deaa05"}
Oct 30 22:10:20.188584 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:20.188522    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e87cf567f0b711ef607ebe6e72cf58ec093e130fcfafab8a6b1097a28deaa05"
Oct 30 22:10:22.962866 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:22.962825    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v"]
Oct 30 22:10:22.963463 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:22.963446    5087 cpu_manager.go:414] "containerMap: removing container" podUID="493e39ae-056c-4407-be7c-9e2f4b4ac812" containerName="revoke-keys"
Oct 30 22:10:22.963463 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:22.963463    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="493e39ae-056c-4407-be7c-9e2f4b4ac812" containerName="revoke-keys"
Oct 30 22:10:22.963595 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:22.963564    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="493e39ae-056c-4407-be7c-9e2f4b4ac812" containerName="revoke-keys"
Oct 30 22:10:22.967150 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:22.967129    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v"
Oct 30 22:10:23.024096 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:23.024061    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v"]
Oct 30 22:10:23.059904 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:23.059869    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f847f946-690e-4d4c-a6b4-3585f52b31db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v\" (UID: \"f847f946-690e-4d4c-a6b4-3585f52b31db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v"
Oct 30 22:10:23.060096 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:23.059994    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f847f946-690e-4d4c-a6b4-3585f52b31db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v\" (UID: \"f847f946-690e-4d4c-a6b4-3585f52b31db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v"
Oct 30 22:10:23.060096 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:23.060080    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg7bv\" (UniqueName: \"kubernetes.io/projected/f847f946-690e-4d4c-a6b4-3585f52b31db-kube-api-access-mg7bv\") pod \"maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v\" (UID: \"f847f946-690e-4d4c-a6b4-3585f52b31db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v"
Oct 30 22:10:23.068610 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:23.068583    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:10:23.068733 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:23.068653    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:10:23.161787 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:23.161749    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f847f946-690e-4d4c-a6b4-3585f52b31db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v\" (UID: \"f847f946-690e-4d4c-a6b4-3585f52b31db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v"
Oct 30 22:10:23.161982 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:23.161843    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f847f946-690e-4d4c-a6b4-3585f52b31db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v\" (UID: \"f847f946-690e-4d4c-a6b4-3585f52b31db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v"
Oct 30 22:10:23.161982 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:23.161890    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg7bv\" (UniqueName: \"kubernetes.io/projected/f847f946-690e-4d4c-a6b4-3585f52b31db-kube-api-access-mg7bv\") pod \"maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v\" (UID: \"f847f946-690e-4d4c-a6b4-3585f52b31db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v"
Oct 30 22:10:23.162701 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:23.162649    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f847f946-690e-4d4c-a6b4-3585f52b31db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v\" (UID: \"f847f946-690e-4d4c-a6b4-3585f52b31db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v"
Oct 30 22:10:23.164362 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:23.164322    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f847f946-690e-4d4c-a6b4-3585f52b31db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v\" (UID: \"f847f946-690e-4d4c-a6b4-3585f52b31db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v"
Oct 30 22:10:23.218414 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:23.218316    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg7bv\" (UniqueName: \"kubernetes.io/projected/f847f946-690e-4d4c-a6b4-3585f52b31db-kube-api-access-mg7bv\") pod \"maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v\" (UID: \"f847f946-690e-4d4c-a6b4-3585f52b31db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v"
Oct 30 22:10:23.281612 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:23.281576    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v"
Oct 30 22:10:23.677385 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:23.673533    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v"]
Oct 30 22:10:23.691070 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:10:23.691022    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf847f946_690e_4d4c_a6b4_3585f52b31db.slice/crio-7e9ad2281e1b0f0bf15cb329d5c2e4e7451993f63a1abd49ec0a0e00ff936784 WatchSource:0}: Error finding container 7e9ad2281e1b0f0bf15cb329d5c2e4e7451993f63a1abd49ec0a0e00ff936784: Status 404 returned error can't find the container with id 7e9ad2281e1b0f0bf15cb329d5c2e4e7451993f63a1abd49ec0a0e00ff936784
Oct 30 22:10:24.222542 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:24.222457    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v" event={"ID":"f847f946-690e-4d4c-a6b4-3585f52b31db","Type":"ContainerStarted","Data":"4df97a8191b4920276f9b54cb1ece64226631132e0a031690ec0189def5bcae8"}
Oct 30 22:10:24.222542 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:24.222507    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v" event={"ID":"f847f946-690e-4d4c-a6b4-3585f52b31db","Type":"ContainerStarted","Data":"7e9ad2281e1b0f0bf15cb329d5c2e4e7451993f63a1abd49ec0a0e00ff936784"}
Oct 30 22:10:24.326383 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:24.326335    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1db768-5z5fd"]
Oct 30 22:10:24.330257 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:24.330241    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1db768-5z5fd"
Oct 30 22:10:24.336706 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:24.336657    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v" podStartSLOduration=2.33664252 podStartE2EDuration="2.33664252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:24.335768546 +0000 UTC m=+2218.810272526" watchObservedRunningTime="2026-10-30 22:10:24.33664252 +0000 UTC m=+2218.811146501"
Oct 30 22:10:24.388136 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:24.388107    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1db768-5z5fd"]
Oct 30 22:10:24.482077 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:24.482042    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c90f6ae-36df-4896-afa5-8859e01044c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-1db768-5z5fd\" (UID: \"1c90f6ae-36df-4896-afa5-8859e01044c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1db768-5z5fd"
Oct 30 22:10:24.482277 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:24.482206    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jw7p\" (UniqueName: \"kubernetes.io/projected/1c90f6ae-36df-4896-afa5-8859e01044c0-kube-api-access-5jw7p\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-1db768-5z5fd\" (UID: \"1c90f6ae-36df-4896-afa5-8859e01044c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1db768-5z5fd"
Oct 30 22:10:24.482341 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:24.482325    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c90f6ae-36df-4896-afa5-8859e01044c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-1db768-5z5fd\" (UID: \"1c90f6ae-36df-4896-afa5-8859e01044c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1db768-5z5fd"
Oct 30 22:10:24.583954 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:24.583919    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c90f6ae-36df-4896-afa5-8859e01044c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-1db768-5z5fd\" (UID: \"1c90f6ae-36df-4896-afa5-8859e01044c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1db768-5z5fd"
Oct 30 22:10:24.584116 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:24.584005    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c90f6ae-36df-4896-afa5-8859e01044c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-1db768-5z5fd\" (UID: \"1c90f6ae-36df-4896-afa5-8859e01044c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1db768-5z5fd"
Oct 30 22:10:24.584179 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:24.584138    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jw7p\" (UniqueName: \"kubernetes.io/projected/1c90f6ae-36df-4896-afa5-8859e01044c0-kube-api-access-5jw7p\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-1db768-5z5fd\" (UID: \"1c90f6ae-36df-4896-afa5-8859e01044c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1db768-5z5fd"
Oct 30 22:10:24.585092 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:24.585063    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c90f6ae-36df-4896-afa5-8859e01044c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-1db768-5z5fd\" (UID: \"1c90f6ae-36df-4896-afa5-8859e01044c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1db768-5z5fd"
Oct 30 22:10:24.586898 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:24.586874    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c90f6ae-36df-4896-afa5-8859e01044c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-1db768-5z5fd\" (UID: \"1c90f6ae-36df-4896-afa5-8859e01044c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1db768-5z5fd"
Oct 30 22:10:24.641090 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:24.641059    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jw7p\" (UniqueName: \"kubernetes.io/projected/1c90f6ae-36df-4896-afa5-8859e01044c0-kube-api-access-5jw7p\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-1db768-5z5fd\" (UID: \"1c90f6ae-36df-4896-afa5-8859e01044c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1db768-5z5fd"
Oct 30 22:10:24.645978 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:24.645748    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1db768-5z5fd"
Oct 30 22:10:24.792181 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:24.790847    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"]
Oct 30 22:10:24.795253 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:24.795219    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:24.853874 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:24.853844    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9da607-openshift-default-dockercfg-5hn82\""
Oct 30 22:10:24.895252 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:24.895208    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-istio-data\") pod \"e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") " pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:24.895532 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:24.895324    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4r9g\" (UniqueName: \"kubernetes.io/projected/a6676e8e-cec6-431d-8a80-ae56c9843093-kube-api-access-d4r9g\") pod \"e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") " pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:24.895532 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:24.895399    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-workload-socket\") pod \"e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") " pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:24.895532 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:24.895483    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6676e8e-cec6-431d-8a80-ae56c9843093-istio-token\") pod \"e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") " pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:24.895532 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:24.895510    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6676e8e-cec6-431d-8a80-ae56c9843093-istiod-ca-cert\") pod \"e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") " pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:24.895767 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:24.895558    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-istio-envoy\") pod \"e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") " pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:24.895767 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:24.895601    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-credential-socket\") pod \"e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") " pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:24.895767 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:24.895639    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6676e8e-cec6-431d-8a80-ae56c9843093-istio-podinfo\") pod \"e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") " pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:24.895767 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:24.895675    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-workload-certs\") pod \"e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") " pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:24.925518 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:24.925413    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"]
Oct 30 22:10:25.008564 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:25.002960    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-workload-certs\") pod \"e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") " pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:25.008564 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:25.003077    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-istio-data\") pod \"e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") " pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:25.008564 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:25.003174    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4r9g\" (UniqueName: \"kubernetes.io/projected/a6676e8e-cec6-431d-8a80-ae56c9843093-kube-api-access-d4r9g\") pod \"e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") " pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:25.008564 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:25.003218    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-workload-socket\") pod \"e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") " pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:25.008564 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:25.003285    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6676e8e-cec6-431d-8a80-ae56c9843093-istio-token\") pod \"e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") " pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:25.008564 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:25.003311    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6676e8e-cec6-431d-8a80-ae56c9843093-istiod-ca-cert\") pod \"e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") " pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:25.008564 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:25.003384    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-istio-envoy\") pod \"e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") " pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:25.008564 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:25.003429    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-credential-socket\") pod \"e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") " pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:25.008564 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:25.003471    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6676e8e-cec6-431d-8a80-ae56c9843093-istio-podinfo\") pod \"e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") " pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:25.008564 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:25.005137    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6676e8e-cec6-431d-8a80-ae56c9843093-istiod-ca-cert\") pod \"e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") " pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:25.008564 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:25.006782    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-workload-socket\") pod \"e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") " pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:25.008564 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:25.007164    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-workload-certs\") pod \"e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") " pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:25.008564 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:25.007446    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-istio-data\") pod \"e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") " pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:25.008564 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:25.007738    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-credential-socket\") pod \"e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") " pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:25.008564 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:25.008325    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6676e8e-cec6-431d-8a80-ae56c9843093-istio-podinfo\") pod \"e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") " pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:25.009587 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:25.009565    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-istio-envoy\") pod \"e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") " pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:25.065879 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:25.065842    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4r9g\" (UniqueName: \"kubernetes.io/projected/a6676e8e-cec6-431d-8a80-ae56c9843093-kube-api-access-d4r9g\") pod \"e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") " pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:25.067009 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:25.066960    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6676e8e-cec6-431d-8a80-ae56c9843093-istio-token\") pod \"e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") " pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:25.075147 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:25.074909    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1db768-5z5fd"]
Oct 30 22:10:25.077624 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:10:25.077596    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c90f6ae_36df_4896_afa5_8859e01044c0.slice/crio-3e73afbc13a57e7e33dcd28db8b61053f539aee931ef85c783257c970aceb06c WatchSource:0}: Error finding container 3e73afbc13a57e7e33dcd28db8b61053f539aee931ef85c783257c970aceb06c: Status 404 returned error can't find the container with id 3e73afbc13a57e7e33dcd28db8b61053f539aee931ef85c783257c970aceb06c
Oct 30 22:10:25.228229 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:25.228198    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:25.231218 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:25.231186    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1db768-5z5fd" event={"ID":"1c90f6ae-36df-4896-afa5-8859e01044c0","Type":"ContainerStarted","Data":"3e73afbc13a57e7e33dcd28db8b61053f539aee931ef85c783257c970aceb06c"}
Oct 30 22:10:25.232992 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:25.232966    5087 generic.go:356] "Generic (PLEG): container finished" podID="f847f946-690e-4d4c-a6b4-3585f52b31db" containerID="4df97a8191b4920276f9b54cb1ece64226631132e0a031690ec0189def5bcae8" exitCode=0
Oct 30 22:10:25.233085 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:25.233040    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v" event={"ID":"f847f946-690e-4d4c-a6b4-3585f52b31db","Type":"ContainerDied","Data":"4df97a8191b4920276f9b54cb1ece64226631132e0a031690ec0189def5bcae8"}
Oct 30 22:10:25.680797 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:25.680770    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"]
Oct 30 22:10:25.682518 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:10:25.682493    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6676e8e_cec6_431d_8a80_ae56c9843093.slice/crio-a93c1c7227b0ec2a2ad31601972238d96ecba19bf60e84fc4d518e7d4be4ad97 WatchSource:0}: Error finding container a93c1c7227b0ec2a2ad31601972238d96ecba19bf60e84fc4d518e7d4be4ad97: Status 404 returned error can't find the container with id a93c1c7227b0ec2a2ad31601972238d96ecba19bf60e84fc4d518e7d4be4ad97
Oct 30 22:10:25.685785 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:25.685752    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:10:25.685865 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:25.685815    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:10:25.685865 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:25.685844    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:10:26.242732 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:26.242695    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42" event={"ID":"a6676e8e-cec6-431d-8a80-ae56c9843093","Type":"ContainerStarted","Data":"f767459022cd88fdbd9239afc6d4af710065bab5b3b1a5871a19dfb45e388d3c"}
Oct 30 22:10:26.243288 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:26.242801    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42" event={"ID":"a6676e8e-cec6-431d-8a80-ae56c9843093","Type":"ContainerStarted","Data":"a93c1c7227b0ec2a2ad31601972238d96ecba19bf60e84fc4d518e7d4be4ad97"}
Oct 30 22:10:26.244713 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:26.244682    5087 generic.go:356] "Generic (PLEG): container finished" podID="1c90f6ae-36df-4896-afa5-8859e01044c0" containerID="69b9a962ae44dd29444fac58a57b945627f3fadba7cf4af56b15fc767731fc6b" exitCode=0
Oct 30 22:10:26.244859 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:26.244717    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1db768-5z5fd" event={"ID":"1c90f6ae-36df-4896-afa5-8859e01044c0","Type":"ContainerDied","Data":"69b9a962ae44dd29444fac58a57b945627f3fadba7cf4af56b15fc767731fc6b"}
Oct 30 22:10:26.364381 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:26.362826    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42" podStartSLOduration=2.362808703 podStartE2EDuration="2.362808703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:26.361156079 +0000 UTC m=+2220.835660070" watchObservedRunningTime="2026-10-30 22:10:26.362808703 +0000 UTC m=+2220.837312682"
Oct 30 22:10:26.436531 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:26.436502    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v"
Oct 30 22:10:26.523901 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:26.523800    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f847f946-690e-4d4c-a6b4-3585f52b31db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f847f946-690e-4d4c-a6b4-3585f52b31db-maas-api-openshift-ca\") pod \"f847f946-690e-4d4c-a6b4-3585f52b31db\" (UID: \"f847f946-690e-4d4c-a6b4-3585f52b31db\") "
Oct 30 22:10:26.523901 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:26.523865    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f847f946-690e-4d4c-a6b4-3585f52b31db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f847f946-690e-4d4c-a6b4-3585f52b31db-maas-api-xks-ca\") pod \"f847f946-690e-4d4c-a6b4-3585f52b31db\" (UID: \"f847f946-690e-4d4c-a6b4-3585f52b31db\") "
Oct 30 22:10:26.524155 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:26.523913    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f847f946-690e-4d4c-a6b4-3585f52b31db-kube-api-access-mg7bv\" (UniqueName: \"kubernetes.io/projected/f847f946-690e-4d4c-a6b4-3585f52b31db-kube-api-access-mg7bv\") pod \"f847f946-690e-4d4c-a6b4-3585f52b31db\" (UID: \"f847f946-690e-4d4c-a6b4-3585f52b31db\") "
Oct 30 22:10:26.524898 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:26.524870    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f847f946-690e-4d4c-a6b4-3585f52b31db-maas-api-openshift-ca" pod "f847f946-690e-4d4c-a6b4-3585f52b31db" (UID: "f847f946-690e-4d4c-a6b4-3585f52b31db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:26.529408 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:26.527233    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f847f946-690e-4d4c-a6b4-3585f52b31db-kube-api-access-mg7bv" pod "f847f946-690e-4d4c-a6b4-3585f52b31db" (UID: "f847f946-690e-4d4c-a6b4-3585f52b31db"). InnerVolumeSpecName "kube-api-access-mg7bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:26.529408 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:26.527526    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f847f946-690e-4d4c-a6b4-3585f52b31db-maas-api-xks-ca" pod "f847f946-690e-4d4c-a6b4-3585f52b31db" (UID: "f847f946-690e-4d4c-a6b4-3585f52b31db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:26.626283 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:26.626246    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f847f946-690e-4d4c-a6b4-3585f52b31db-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:26.626283 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:26.626280    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f847f946-690e-4d4c-a6b4-3585f52b31db-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:26.626492 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:26.626295    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg7bv\" (UniqueName: \"kubernetes.io/projected/f847f946-690e-4d4c-a6b4-3585f52b31db-kube-api-access-mg7bv\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:27.229285 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:27.229252    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:27.235330 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:27.235304    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:27.254600 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:27.254572    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v"
Oct 30 22:10:27.254600 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:27.254586    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v" event={"ID":"f847f946-690e-4d4c-a6b4-3585f52b31db","Type":"ContainerDied","Data":"7e9ad2281e1b0f0bf15cb329d5c2e4e7451993f63a1abd49ec0a0e00ff936784"}
Oct 30 22:10:27.255049 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:27.254636    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e9ad2281e1b0f0bf15cb329d5c2e4e7451993f63a1abd49ec0a0e00ff936784"
Oct 30 22:10:27.255113 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:27.255081    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:27.256219 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:27.256200    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:10:27.421635 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:27.421614    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1db768-5z5fd"
Oct 30 22:10:27.537529 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:27.537488    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c90f6ae-36df-4896-afa5-8859e01044c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c90f6ae-36df-4896-afa5-8859e01044c0-maas-api-xks-ca\") pod \"1c90f6ae-36df-4896-afa5-8859e01044c0\" (UID: \"1c90f6ae-36df-4896-afa5-8859e01044c0\") "
Oct 30 22:10:27.537764 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:27.537657    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c90f6ae-36df-4896-afa5-8859e01044c0-kube-api-access-5jw7p\" (UniqueName: \"kubernetes.io/projected/1c90f6ae-36df-4896-afa5-8859e01044c0-kube-api-access-5jw7p\") pod \"1c90f6ae-36df-4896-afa5-8859e01044c0\" (UID: \"1c90f6ae-36df-4896-afa5-8859e01044c0\") "
Oct 30 22:10:27.537764 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:27.537751    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c90f6ae-36df-4896-afa5-8859e01044c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c90f6ae-36df-4896-afa5-8859e01044c0-maas-api-openshift-ca\") pod \"1c90f6ae-36df-4896-afa5-8859e01044c0\" (UID: \"1c90f6ae-36df-4896-afa5-8859e01044c0\") "
Oct 30 22:10:27.538609 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:27.538574    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c90f6ae-36df-4896-afa5-8859e01044c0-maas-api-openshift-ca" pod "1c90f6ae-36df-4896-afa5-8859e01044c0" (UID: "1c90f6ae-36df-4896-afa5-8859e01044c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:27.540550 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:27.540521    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c90f6ae-36df-4896-afa5-8859e01044c0-maas-api-xks-ca" pod "1c90f6ae-36df-4896-afa5-8859e01044c0" (UID: "1c90f6ae-36df-4896-afa5-8859e01044c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:27.541011 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:27.540987    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c90f6ae-36df-4896-afa5-8859e01044c0-kube-api-access-5jw7p" pod "1c90f6ae-36df-4896-afa5-8859e01044c0" (UID: "1c90f6ae-36df-4896-afa5-8859e01044c0"). InnerVolumeSpecName "kube-api-access-5jw7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:27.639537 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:27.639508    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jw7p\" (UniqueName: \"kubernetes.io/projected/1c90f6ae-36df-4896-afa5-8859e01044c0-kube-api-access-5jw7p\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:27.639760 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:27.639744    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c90f6ae-36df-4896-afa5-8859e01044c0-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:27.639859 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:27.639848    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c90f6ae-36df-4896-afa5-8859e01044c0-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:10:28.286263 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:28.283448    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1db768-5z5fd"
Oct 30 22:10:28.286263 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:28.283584    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1db768-5z5fd" event={"ID":"1c90f6ae-36df-4896-afa5-8859e01044c0","Type":"ContainerDied","Data":"3e73afbc13a57e7e33dcd28db8b61053f539aee931ef85c783257c970aceb06c"}
Oct 30 22:10:28.286263 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:28.283630    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e73afbc13a57e7e33dcd28db8b61053f539aee931ef85c783257c970aceb06c"
Oct 30 22:10:42.998414 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:42.998369    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6cd200-gzbct"]
Oct 30 22:10:43.047440 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:43.047410    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-6cd200-gzbct"]
Oct 30 22:10:43.710140 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:43.710095    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="493e39ae-056c-4407-be7c-9e2f4b4ac812" path="/var/lib/kubelet/pods/493e39ae-056c-4407-be7c-9e2f4b4ac812/volumes"
Oct 30 22:10:47.169043 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:10:47.168988    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:10:49.582960 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:49.582926    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b"]
Oct 30 22:10:49.632141 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:49.632106    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-485d1e-vct6b"]
Oct 30 22:10:49.710107 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:49.710076    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07358162-aa30-4455-9759-1e7fcf1c4871" path="/var/lib/kubelet/pods/07358162-aa30-4455-9759-1e7fcf1c4871/volumes"
Oct 30 22:10:56.085356 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:56.085307    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr"]
Oct 30 22:10:56.135057 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:56.135021    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-2qjgr"]
Oct 30 22:10:57.709243 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:10:57.709210    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18b3281c-0a59-4598-b4cc-b3a4565b0c01" path="/var/lib/kubelet/pods/18b3281c-0a59-4598-b4cc-b3a4565b0c01/volumes"
Oct 30 22:11:05.711316 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:05.711284    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj"]
Oct 30 22:11:05.711795 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:05.711487    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj" podUID="7372b3a2-49ea-42b8-9feb-48b67269aaad" containerName="maas-api" containerID="cri-o://9d2b96e1dd3ff69d47019425297b7fb53d8bc77e4b765bb014b8702435d903bd" gracePeriod=30
Oct 30 22:11:06.029812 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:06.029774    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj"
Oct 30 22:11:06.065814 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:06.065773    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7372b3a2-49ea-42b8-9feb-48b67269aaad-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7372b3a2-49ea-42b8-9feb-48b67269aaad-metrics-certs\") pod \"7372b3a2-49ea-42b8-9feb-48b67269aaad\" (UID: \"7372b3a2-49ea-42b8-9feb-48b67269aaad\") "
Oct 30 22:11:06.066038 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:06.065913    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7372b3a2-49ea-42b8-9feb-48b67269aaad-kube-api-access-fblzv\" (UniqueName: \"kubernetes.io/projected/7372b3a2-49ea-42b8-9feb-48b67269aaad-kube-api-access-fblzv\") pod \"7372b3a2-49ea-42b8-9feb-48b67269aaad\" (UID: \"7372b3a2-49ea-42b8-9feb-48b67269aaad\") "
Oct 30 22:11:06.066139 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:06.066120    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7372b3a2-49ea-42b8-9feb-48b67269aaad-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7372b3a2-49ea-42b8-9feb-48b67269aaad-maas-api-tls\") pod \"7372b3a2-49ea-42b8-9feb-48b67269aaad\" (UID: \"7372b3a2-49ea-42b8-9feb-48b67269aaad\") "
Oct 30 22:11:06.066209 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:06.066167    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7372b3a2-49ea-42b8-9feb-48b67269aaad-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7372b3a2-49ea-42b8-9feb-48b67269aaad-openshift-service-ca\") pod \"7372b3a2-49ea-42b8-9feb-48b67269aaad\" (UID: \"7372b3a2-49ea-42b8-9feb-48b67269aaad\") "
Oct 30 22:11:06.066751 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:06.066699    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7372b3a2-49ea-42b8-9feb-48b67269aaad-openshift-service-ca" pod "7372b3a2-49ea-42b8-9feb-48b67269aaad" (UID: "7372b3a2-49ea-42b8-9feb-48b67269aaad"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:06.068467 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:06.068439    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7372b3a2-49ea-42b8-9feb-48b67269aaad-maas-api-tls" pod "7372b3a2-49ea-42b8-9feb-48b67269aaad" (UID: "7372b3a2-49ea-42b8-9feb-48b67269aaad"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:06.068701 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:06.068669    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7372b3a2-49ea-42b8-9feb-48b67269aaad-metrics-certs" pod "7372b3a2-49ea-42b8-9feb-48b67269aaad" (UID: "7372b3a2-49ea-42b8-9feb-48b67269aaad"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:06.068795 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:06.068715    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7372b3a2-49ea-42b8-9feb-48b67269aaad-kube-api-access-fblzv" pod "7372b3a2-49ea-42b8-9feb-48b67269aaad" (UID: "7372b3a2-49ea-42b8-9feb-48b67269aaad"). InnerVolumeSpecName "kube-api-access-fblzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:06.168039 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:06.167999    5087 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7372b3a2-49ea-42b8-9feb-48b67269aaad-openshift-service-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:06.168039 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:06.168033    5087 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7372b3a2-49ea-42b8-9feb-48b67269aaad-metrics-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:06.168233 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:06.168046    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fblzv\" (UniqueName: \"kubernetes.io/projected/7372b3a2-49ea-42b8-9feb-48b67269aaad-kube-api-access-fblzv\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:06.168233 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:06.168060    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7372b3a2-49ea-42b8-9feb-48b67269aaad-maas-api-tls\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:06.567071 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:06.567038    5087 generic.go:356] "Generic (PLEG): container finished" podID="7372b3a2-49ea-42b8-9feb-48b67269aaad" containerID="9d2b96e1dd3ff69d47019425297b7fb53d8bc77e4b765bb014b8702435d903bd" exitCode=0
Oct 30 22:11:06.567333 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:06.567093    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj"
Oct 30 22:11:06.567333 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:06.567136    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj" event={"ID":"7372b3a2-49ea-42b8-9feb-48b67269aaad","Type":"ContainerDied","Data":"9d2b96e1dd3ff69d47019425297b7fb53d8bc77e4b765bb014b8702435d903bd"}
Oct 30 22:11:06.567333 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:06.567209    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj" event={"ID":"7372b3a2-49ea-42b8-9feb-48b67269aaad","Type":"ContainerDied","Data":"86ffbd240b2f3922253540edc22119820c3b16cfdd9ef87f5449d28b4786e001"}
Oct 30 22:11:06.567333 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:06.567236    5087 scope.go:122] "RemoveContainer" containerID="9d2b96e1dd3ff69d47019425297b7fb53d8bc77e4b765bb014b8702435d903bd"
Oct 30 22:11:06.579416 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:06.579366    5087 scope.go:122] "RemoveContainer" containerID="9d2b96e1dd3ff69d47019425297b7fb53d8bc77e4b765bb014b8702435d903bd"
Oct 30 22:11:06.579696 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:11:06.579672    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d2b96e1dd3ff69d47019425297b7fb53d8bc77e4b765bb014b8702435d903bd\": container with ID starting with 9d2b96e1dd3ff69d47019425297b7fb53d8bc77e4b765bb014b8702435d903bd not found: ID does not exist" containerID="9d2b96e1dd3ff69d47019425297b7fb53d8bc77e4b765bb014b8702435d903bd"
Oct 30 22:11:06.579780 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:06.579708    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d2b96e1dd3ff69d47019425297b7fb53d8bc77e4b765bb014b8702435d903bd"} err="failed to get container status \"9d2b96e1dd3ff69d47019425297b7fb53d8bc77e4b765bb014b8702435d903bd\": rpc error: code = NotFound desc = could not find container \"9d2b96e1dd3ff69d47019425297b7fb53d8bc77e4b765bb014b8702435d903bd\": container with ID starting with 9d2b96e1dd3ff69d47019425297b7fb53d8bc77e4b765bb014b8702435d903bd not found: ID does not exist"
Oct 30 22:11:06.738278 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:06.738245    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj"]
Oct 30 22:11:06.741228 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:06.741205    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-9c8c4d-f645d459b-swmnj"]
Oct 30 22:11:07.709337 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:07.709307    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7372b3a2-49ea-42b8-9feb-48b67269aaad" path="/var/lib/kubelet/pods/7372b3a2-49ea-42b8-9feb-48b67269aaad/volumes"
Oct 30 22:11:08.102680 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:08.102639    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c8c4d-mdhnv"]
Oct 30 22:11:08.105575 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:08.105555    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9c8c4d-mdhnv"]
Oct 30 22:11:08.505036 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:08.505003    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dda9f5-v2pzf"]
Oct 30 22:11:08.505641 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:08.505626    5087 cpu_manager.go:414] "containerMap: removing container" podUID="1c90f6ae-36df-4896-afa5-8859e01044c0" containerName="revoke-keys"
Oct 30 22:11:08.505694 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:08.505644    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c90f6ae-36df-4896-afa5-8859e01044c0" containerName="revoke-keys"
Oct 30 22:11:08.505694 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:08.505669    5087 cpu_manager.go:414] "containerMap: removing container" podUID="7372b3a2-49ea-42b8-9feb-48b67269aaad" containerName="maas-api"
Oct 30 22:11:08.505694 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:08.505675    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7372b3a2-49ea-42b8-9feb-48b67269aaad" containerName="maas-api"
Oct 30 22:11:08.505794 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:08.505698    5087 cpu_manager.go:414] "containerMap: removing container" podUID="f847f946-690e-4d4c-a6b4-3585f52b31db" containerName="revoke-keys"
Oct 30 22:11:08.505794 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:08.505709    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f847f946-690e-4d4c-a6b4-3585f52b31db" containerName="revoke-keys"
Oct 30 22:11:08.505858 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:08.505800    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="7372b3a2-49ea-42b8-9feb-48b67269aaad" containerName="maas-api"
Oct 30 22:11:08.505858 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:08.505812    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="f847f946-690e-4d4c-a6b4-3585f52b31db" containerName="revoke-keys"
Oct 30 22:11:08.505858 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:08.505821    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c90f6ae-36df-4896-afa5-8859e01044c0" containerName="revoke-keys"
Oct 30 22:11:08.510478 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:08.510460    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dda9f5-v2pzf"
Oct 30 22:11:08.562238 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:08.562215    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:11:08.565833 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:08.565807    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dda9f5-v2pzf"]
Oct 30 22:11:08.610517 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:08.610489    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:11:08.704745 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:08.704709    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1269e40-c1fe-4166-ad83-6f07a8c95169-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-dda9f5-v2pzf\" (UID: \"a1269e40-c1fe-4166-ad83-6f07a8c95169\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dda9f5-v2pzf"
Oct 30 22:11:08.704956 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:08.704889    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1269e40-c1fe-4166-ad83-6f07a8c95169-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-dda9f5-v2pzf\" (UID: \"a1269e40-c1fe-4166-ad83-6f07a8c95169\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dda9f5-v2pzf"
Oct 30 22:11:08.705147 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:08.705120    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgqhz\" (UniqueName: \"kubernetes.io/projected/a1269e40-c1fe-4166-ad83-6f07a8c95169-kube-api-access-sgqhz\") pod \"maas-api-revoke-keys-e2e-worker-w0-dda9f5-v2pzf\" (UID: \"a1269e40-c1fe-4166-ad83-6f07a8c95169\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dda9f5-v2pzf"
Oct 30 22:11:08.805938 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:08.805856    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1269e40-c1fe-4166-ad83-6f07a8c95169-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-dda9f5-v2pzf\" (UID: \"a1269e40-c1fe-4166-ad83-6f07a8c95169\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dda9f5-v2pzf"
Oct 30 22:11:08.806113 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:08.805969    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgqhz\" (UniqueName: \"kubernetes.io/projected/a1269e40-c1fe-4166-ad83-6f07a8c95169-kube-api-access-sgqhz\") pod \"maas-api-revoke-keys-e2e-worker-w0-dda9f5-v2pzf\" (UID: \"a1269e40-c1fe-4166-ad83-6f07a8c95169\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dda9f5-v2pzf"
Oct 30 22:11:08.806113 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:08.806000    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1269e40-c1fe-4166-ad83-6f07a8c95169-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-dda9f5-v2pzf\" (UID: \"a1269e40-c1fe-4166-ad83-6f07a8c95169\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dda9f5-v2pzf"
Oct 30 22:11:08.806693 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:08.806670    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1269e40-c1fe-4166-ad83-6f07a8c95169-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-dda9f5-v2pzf\" (UID: \"a1269e40-c1fe-4166-ad83-6f07a8c95169\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dda9f5-v2pzf"
Oct 30 22:11:08.808456 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:08.808433    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1269e40-c1fe-4166-ad83-6f07a8c95169-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-dda9f5-v2pzf\" (UID: \"a1269e40-c1fe-4166-ad83-6f07a8c95169\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dda9f5-v2pzf"
Oct 30 22:11:08.863649 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:08.863619    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgqhz\" (UniqueName: \"kubernetes.io/projected/a1269e40-c1fe-4166-ad83-6f07a8c95169-kube-api-access-sgqhz\") pod \"maas-api-revoke-keys-e2e-worker-w0-dda9f5-v2pzf\" (UID: \"a1269e40-c1fe-4166-ad83-6f07a8c95169\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dda9f5-v2pzf"
Oct 30 22:11:09.126718 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:09.126643    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dda9f5-v2pzf"
Oct 30 22:11:09.505768 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:09.505739    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dda9f5-v2pzf"]
Oct 30 22:11:09.506876 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:11:09.506847    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1269e40_c1fe_4166_ad83_6f07a8c95169.slice/crio-41968fa71191ae41ab47c3fcf31891631d34165cb407a6a6f6553c6c7607c201 WatchSource:0}: Error finding container 41968fa71191ae41ab47c3fcf31891631d34165cb407a6a6f6553c6c7607c201: Status 404 returned error can't find the container with id 41968fa71191ae41ab47c3fcf31891631d34165cb407a6a6f6553c6c7607c201
Oct 30 22:11:09.611624 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:09.611580    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dda9f5-v2pzf" event={"ID":"a1269e40-c1fe-4166-ad83-6f07a8c95169","Type":"ContainerStarted","Data":"41968fa71191ae41ab47c3fcf31891631d34165cb407a6a6f6553c6c7607c201"}
Oct 30 22:11:09.709727 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:09.709700    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5801207-d12c-435e-b488-362e01e8c6d4" path="/var/lib/kubelet/pods/b5801207-d12c-435e-b488-362e01e8c6d4/volumes"
Oct 30 22:11:10.245701 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:10.245668    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dde8a228-xzmw8"]
Oct 30 22:11:10.249712 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:10.249692    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde8a228-xzmw8"
Oct 30 22:11:10.322930 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:10.322900    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hsfz\" (UniqueName: \"kubernetes.io/projected/8cde2a2e-3765-4cd3-a1a8-ac979f272afc-kube-api-access-2hsfz\") pod \"maas-api-revoke-sub-dde8a228-xzmw8\" (UID: \"8cde2a2e-3765-4cd3-a1a8-ac979f272afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde8a228-xzmw8"
Oct 30 22:11:10.323091 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:10.322953    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cde2a2e-3765-4cd3-a1a8-ac979f272afc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dde8a228-xzmw8\" (UID: \"8cde2a2e-3765-4cd3-a1a8-ac979f272afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde8a228-xzmw8"
Oct 30 22:11:10.323091 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:10.323083    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cde2a2e-3765-4cd3-a1a8-ac979f272afc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dde8a228-xzmw8\" (UID: \"8cde2a2e-3765-4cd3-a1a8-ac979f272afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde8a228-xzmw8"
Oct 30 22:11:10.358425 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:10.358377    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dde8a228-xzmw8"]
Oct 30 22:11:10.424419 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:10.424328    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hsfz\" (UniqueName: \"kubernetes.io/projected/8cde2a2e-3765-4cd3-a1a8-ac979f272afc-kube-api-access-2hsfz\") pod \"maas-api-revoke-sub-dde8a228-xzmw8\" (UID: \"8cde2a2e-3765-4cd3-a1a8-ac979f272afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde8a228-xzmw8"
Oct 30 22:11:10.424623 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:10.424468    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cde2a2e-3765-4cd3-a1a8-ac979f272afc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dde8a228-xzmw8\" (UID: \"8cde2a2e-3765-4cd3-a1a8-ac979f272afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde8a228-xzmw8"
Oct 30 22:11:10.424623 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:10.424590    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cde2a2e-3765-4cd3-a1a8-ac979f272afc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dde8a228-xzmw8\" (UID: \"8cde2a2e-3765-4cd3-a1a8-ac979f272afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde8a228-xzmw8"
Oct 30 22:11:10.425182 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:10.425156    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cde2a2e-3765-4cd3-a1a8-ac979f272afc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dde8a228-xzmw8\" (UID: \"8cde2a2e-3765-4cd3-a1a8-ac979f272afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde8a228-xzmw8"
Oct 30 22:11:10.427216 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:10.427194    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cde2a2e-3765-4cd3-a1a8-ac979f272afc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dde8a228-xzmw8\" (UID: \"8cde2a2e-3765-4cd3-a1a8-ac979f272afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde8a228-xzmw8"
Oct 30 22:11:10.481548 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:10.481516    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hsfz\" (UniqueName: \"kubernetes.io/projected/8cde2a2e-3765-4cd3-a1a8-ac979f272afc-kube-api-access-2hsfz\") pod \"maas-api-revoke-sub-dde8a228-xzmw8\" (UID: \"8cde2a2e-3765-4cd3-a1a8-ac979f272afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde8a228-xzmw8"
Oct 30 22:11:10.564936 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:10.564904    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde8a228-xzmw8"
Oct 30 22:11:10.621208 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:10.621176    5087 generic.go:356] "Generic (PLEG): container finished" podID="a1269e40-c1fe-4166-ad83-6f07a8c95169" containerID="70bc5dd5d10d5c0bef0dc603aa17dc5ba95f58cf93ad2ec848b7082b73258cbd" exitCode=0
Oct 30 22:11:10.621409 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:10.621258    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dda9f5-v2pzf" event={"ID":"a1269e40-c1fe-4166-ad83-6f07a8c95169","Type":"ContainerDied","Data":"70bc5dd5d10d5c0bef0dc603aa17dc5ba95f58cf93ad2ec848b7082b73258cbd"}
Oct 30 22:11:10.948694 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:10.948669    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dde8a228-xzmw8"]
Oct 30 22:11:10.949644 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:11:10.949609    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cde2a2e_3765_4cd3_a1a8_ac979f272afc.slice/crio-80ad20e926e0cd5d32e1c1739b97c902c8d18dc89d7708ae4757f48d16c7ba68 WatchSource:0}: Error finding container 80ad20e926e0cd5d32e1c1739b97c902c8d18dc89d7708ae4757f48d16c7ba68: Status 404 returned error can't find the container with id 80ad20e926e0cd5d32e1c1739b97c902c8d18dc89d7708ae4757f48d16c7ba68
Oct 30 22:11:11.629400 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:11.629295    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde8a228-xzmw8" event={"ID":"8cde2a2e-3765-4cd3-a1a8-ac979f272afc","Type":"ContainerStarted","Data":"f461d9615c419d3c16320c9848c9ad1048833e923609186944f3bfc146af8557"}
Oct 30 22:11:11.629400 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:11.629375    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde8a228-xzmw8" event={"ID":"8cde2a2e-3765-4cd3-a1a8-ac979f272afc","Type":"ContainerStarted","Data":"80ad20e926e0cd5d32e1c1739b97c902c8d18dc89d7708ae4757f48d16c7ba68"}
Oct 30 22:11:11.746111 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:11.746049    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde8a228-xzmw8" podStartSLOduration=1.746030077 podStartE2EDuration="1.746030077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:11.743375677 +0000 UTC m=+2266.217879657" watchObservedRunningTime="2026-10-30 22:11:11.746030077 +0000 UTC m=+2266.220534059"
Oct 30 22:11:11.795163 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:11.795139    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dda9f5-v2pzf"
Oct 30 22:11:11.845184 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:11.845155    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1269e40-c1fe-4166-ad83-6f07a8c95169-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1269e40-c1fe-4166-ad83-6f07a8c95169-maas-api-xks-ca\") pod \"a1269e40-c1fe-4166-ad83-6f07a8c95169\" (UID: \"a1269e40-c1fe-4166-ad83-6f07a8c95169\") "
Oct 30 22:11:11.845392 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:11.845225    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1269e40-c1fe-4166-ad83-6f07a8c95169-kube-api-access-sgqhz\" (UniqueName: \"kubernetes.io/projected/a1269e40-c1fe-4166-ad83-6f07a8c95169-kube-api-access-sgqhz\") pod \"a1269e40-c1fe-4166-ad83-6f07a8c95169\" (UID: \"a1269e40-c1fe-4166-ad83-6f07a8c95169\") "
Oct 30 22:11:11.845392 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:11.845270    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1269e40-c1fe-4166-ad83-6f07a8c95169-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1269e40-c1fe-4166-ad83-6f07a8c95169-maas-api-openshift-ca\") pod \"a1269e40-c1fe-4166-ad83-6f07a8c95169\" (UID: \"a1269e40-c1fe-4166-ad83-6f07a8c95169\") "
Oct 30 22:11:11.845728 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:11.845696    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1269e40-c1fe-4166-ad83-6f07a8c95169-maas-api-openshift-ca" pod "a1269e40-c1fe-4166-ad83-6f07a8c95169" (UID: "a1269e40-c1fe-4166-ad83-6f07a8c95169"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:11.845926 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:11.845909    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1269e40-c1fe-4166-ad83-6f07a8c95169-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:11.847534 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:11.847500    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1269e40-c1fe-4166-ad83-6f07a8c95169-maas-api-xks-ca" pod "a1269e40-c1fe-4166-ad83-6f07a8c95169" (UID: "a1269e40-c1fe-4166-ad83-6f07a8c95169"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:11.847770 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:11.847751    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1269e40-c1fe-4166-ad83-6f07a8c95169-kube-api-access-sgqhz" pod "a1269e40-c1fe-4166-ad83-6f07a8c95169" (UID: "a1269e40-c1fe-4166-ad83-6f07a8c95169"). InnerVolumeSpecName "kube-api-access-sgqhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:11.947282 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:11.947198    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1269e40-c1fe-4166-ad83-6f07a8c95169-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:11.947282 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:11.947238    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgqhz\" (UniqueName: \"kubernetes.io/projected/a1269e40-c1fe-4166-ad83-6f07a8c95169-kube-api-access-sgqhz\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:12.637717 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:12.637670    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dda9f5-v2pzf" event={"ID":"a1269e40-c1fe-4166-ad83-6f07a8c95169","Type":"ContainerDied","Data":"41968fa71191ae41ab47c3fcf31891631d34165cb407a6a6f6553c6c7607c201"}
Oct 30 22:11:12.638231 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:12.637742    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41968fa71191ae41ab47c3fcf31891631d34165cb407a6a6f6553c6c7607c201"
Oct 30 22:11:12.638231 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:12.637689    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dda9f5-v2pzf"
Oct 30 22:11:12.639851 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:12.639822    5087 generic.go:356] "Generic (PLEG): container finished" podID="8cde2a2e-3765-4cd3-a1a8-ac979f272afc" containerID="f461d9615c419d3c16320c9848c9ad1048833e923609186944f3bfc146af8557" exitCode=0
Oct 30 22:11:12.640088 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:12.640044    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde8a228-xzmw8" event={"ID":"8cde2a2e-3765-4cd3-a1a8-ac979f272afc","Type":"ContainerDied","Data":"f461d9615c419d3c16320c9848c9ad1048833e923609186944f3bfc146af8557"}
Oct 30 22:11:12.820278 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:12.820247    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"]
Oct 30 22:11:12.825062 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:12.825037    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"]
Oct 30 22:11:13.822285 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:13.822263    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde8a228-xzmw8"
Oct 30 22:11:13.873884 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:13.873853    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cde2a2e-3765-4cd3-a1a8-ac979f272afc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cde2a2e-3765-4cd3-a1a8-ac979f272afc-maas-api-xks-ca\") pod \"8cde2a2e-3765-4cd3-a1a8-ac979f272afc\" (UID: \"8cde2a2e-3765-4cd3-a1a8-ac979f272afc\") "
Oct 30 22:11:13.874058 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:13.873902    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cde2a2e-3765-4cd3-a1a8-ac979f272afc-kube-api-access-2hsfz\" (UniqueName: \"kubernetes.io/projected/8cde2a2e-3765-4cd3-a1a8-ac979f272afc-kube-api-access-2hsfz\") pod \"8cde2a2e-3765-4cd3-a1a8-ac979f272afc\" (UID: \"8cde2a2e-3765-4cd3-a1a8-ac979f272afc\") "
Oct 30 22:11:13.874058 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:13.873992    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cde2a2e-3765-4cd3-a1a8-ac979f272afc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cde2a2e-3765-4cd3-a1a8-ac979f272afc-maas-api-openshift-ca\") pod \"8cde2a2e-3765-4cd3-a1a8-ac979f272afc\" (UID: \"8cde2a2e-3765-4cd3-a1a8-ac979f272afc\") "
Oct 30 22:11:13.874485 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:13.874457    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cde2a2e-3765-4cd3-a1a8-ac979f272afc-maas-api-openshift-ca" pod "8cde2a2e-3765-4cd3-a1a8-ac979f272afc" (UID: "8cde2a2e-3765-4cd3-a1a8-ac979f272afc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:13.874760 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:13.874711    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cde2a2e-3765-4cd3-a1a8-ac979f272afc-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:13.876276 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:13.876243    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cde2a2e-3765-4cd3-a1a8-ac979f272afc-maas-api-xks-ca" pod "8cde2a2e-3765-4cd3-a1a8-ac979f272afc" (UID: "8cde2a2e-3765-4cd3-a1a8-ac979f272afc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:13.876408 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:13.876391    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cde2a2e-3765-4cd3-a1a8-ac979f272afc-kube-api-access-2hsfz" pod "8cde2a2e-3765-4cd3-a1a8-ac979f272afc" (UID: "8cde2a2e-3765-4cd3-a1a8-ac979f272afc"). InnerVolumeSpecName "kube-api-access-2hsfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:13.976855 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:13.976768    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cde2a2e-3765-4cd3-a1a8-ac979f272afc-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:13.976855 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:13.976806    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hsfz\" (UniqueName: \"kubernetes.io/projected/8cde2a2e-3765-4cd3-a1a8-ac979f272afc-kube-api-access-2hsfz\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:14.660994 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:14.660962    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde8a228-xzmw8"
Oct 30 22:11:14.661190 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:14.660996    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dde8a228-xzmw8" event={"ID":"8cde2a2e-3765-4cd3-a1a8-ac979f272afc","Type":"ContainerDied","Data":"80ad20e926e0cd5d32e1c1739b97c902c8d18dc89d7708ae4757f48d16c7ba68"}
Oct 30 22:11:14.661190 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:14.661045    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80ad20e926e0cd5d32e1c1739b97c902c8d18dc89d7708ae4757f48d16c7ba68"
Oct 30 22:11:14.838225 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:14.838190    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"]
Oct 30 22:11:14.838845 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:14.838814    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82" podUID="c975ae55-4f25-494b-acd0-7458374253df" containerName="istio-proxy" containerID="cri-o://cf0513a665aab5adf830e5842a34964c8a2609604c7220ea411abf6df9199c55" gracePeriod=30
Oct 30 22:11:17.171625 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:11:17.171583    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:11:17.909417 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:17.909326    5087 patch_prober.go:29] interesting pod/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:11:17.909778 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:17.909740    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82" podUID="c975ae55-4f25-494b-acd0-7458374253df" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:11:18.876564 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:18.876531    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw"]
Oct 30 22:11:18.985608 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:18.985569    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v"]
Oct 30 22:11:20.136949 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.136924    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:11:20.256735 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.256683    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-workload-socket\") pod \"c975ae55-4f25-494b-acd0-7458374253df\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") "
Oct 30 22:11:20.256735 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.256737    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-credential-socket\") pod \"c975ae55-4f25-494b-acd0-7458374253df\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") "
Oct 30 22:11:20.257011 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.256765    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-workload-certs\") pod \"c975ae55-4f25-494b-acd0-7458374253df\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") "
Oct 30 22:11:20.257011 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.256828    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c975ae55-4f25-494b-acd0-7458374253df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c975ae55-4f25-494b-acd0-7458374253df-istiod-ca-cert\") pod \"c975ae55-4f25-494b-acd0-7458374253df\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") "
Oct 30 22:11:20.257011 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.256981    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c975ae55-4f25-494b-acd0-7458374253df-istio-token\" (UniqueName: \"kubernetes.io/projected/c975ae55-4f25-494b-acd0-7458374253df-istio-token\") pod \"c975ae55-4f25-494b-acd0-7458374253df\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") "
Oct 30 22:11:20.257181 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.257020    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c975ae55-4f25-494b-acd0-7458374253df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c975ae55-4f25-494b-acd0-7458374253df-istio-podinfo\") pod \"c975ae55-4f25-494b-acd0-7458374253df\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") "
Oct 30 22:11:20.257181 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.257064    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c975ae55-4f25-494b-acd0-7458374253df-kube-api-access-24swd\" (UniqueName: \"kubernetes.io/projected/c975ae55-4f25-494b-acd0-7458374253df-kube-api-access-24swd\") pod \"c975ae55-4f25-494b-acd0-7458374253df\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") "
Oct 30 22:11:20.257181 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.257141    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-istio-envoy\") pod \"c975ae55-4f25-494b-acd0-7458374253df\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") "
Oct 30 22:11:20.257181 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.257168    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-istio-data\") pod \"c975ae55-4f25-494b-acd0-7458374253df\" (UID: \"c975ae55-4f25-494b-acd0-7458374253df\") "
Oct 30 22:11:20.258599 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.258568    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-workload-socket" pod "c975ae55-4f25-494b-acd0-7458374253df" (UID: "c975ae55-4f25-494b-acd0-7458374253df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:20.258831 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.258659    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c975ae55-4f25-494b-acd0-7458374253df-istiod-ca-cert" pod "c975ae55-4f25-494b-acd0-7458374253df" (UID: "c975ae55-4f25-494b-acd0-7458374253df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:20.258831 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.258823    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-workload-certs" pod "c975ae55-4f25-494b-acd0-7458374253df" (UID: "c975ae55-4f25-494b-acd0-7458374253df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:20.259057 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.259028    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-istio-data" pod "c975ae55-4f25-494b-acd0-7458374253df" (UID: "c975ae55-4f25-494b-acd0-7458374253df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:20.259310 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.259291    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-credential-socket" pod "c975ae55-4f25-494b-acd0-7458374253df" (UID: "c975ae55-4f25-494b-acd0-7458374253df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:20.261682 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.261652    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c975ae55-4f25-494b-acd0-7458374253df-kube-api-access-24swd" pod "c975ae55-4f25-494b-acd0-7458374253df" (UID: "c975ae55-4f25-494b-acd0-7458374253df"). InnerVolumeSpecName "kube-api-access-24swd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:20.262025 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.261979    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c975ae55-4f25-494b-acd0-7458374253df-istio-token" pod "c975ae55-4f25-494b-acd0-7458374253df" (UID: "c975ae55-4f25-494b-acd0-7458374253df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:20.262025 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.262020    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c975ae55-4f25-494b-acd0-7458374253df-istio-podinfo" pod "c975ae55-4f25-494b-acd0-7458374253df" (UID: "c975ae55-4f25-494b-acd0-7458374253df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:20.262495 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.262465    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-istio-envoy" pod "c975ae55-4f25-494b-acd0-7458374253df" (UID: "c975ae55-4f25-494b-acd0-7458374253df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:20.362547 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.362293    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c975ae55-4f25-494b-acd0-7458374253df-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:20.362547 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.362330    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c975ae55-4f25-494b-acd0-7458374253df-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:20.362547 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.362384    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24swd\" (UniqueName: \"kubernetes.io/projected/c975ae55-4f25-494b-acd0-7458374253df-kube-api-access-24swd\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:20.362547 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.362400    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:20.362547 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.362413    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:20.362547 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.362425    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:20.362547 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.362437    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:20.362547 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.362449    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c975ae55-4f25-494b-acd0-7458374253df-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:20.362547 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.362461    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c975ae55-4f25-494b-acd0-7458374253df-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:20.709510 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.709411    5087 generic.go:356] "Generic (PLEG): container finished" podID="c975ae55-4f25-494b-acd0-7458374253df" containerID="cf0513a665aab5adf830e5842a34964c8a2609604c7220ea411abf6df9199c55" exitCode=0
Oct 30 22:11:20.709510 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.709468    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"
Oct 30 22:11:20.709510 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.709495    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82" event={"ID":"c975ae55-4f25-494b-acd0-7458374253df","Type":"ContainerDied","Data":"cf0513a665aab5adf830e5842a34964c8a2609604c7220ea411abf6df9199c55"}
Oct 30 22:11:20.709791 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.709547    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82" event={"ID":"c975ae55-4f25-494b-acd0-7458374253df","Type":"ContainerDied","Data":"23457629204cbdfb787d8bdf87e6adfc1c5c2f2952bea7ea20308de0a4187a18"}
Oct 30 22:11:20.709791 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.709570    5087 scope.go:122] "RemoveContainer" containerID="cf0513a665aab5adf830e5842a34964c8a2609604c7220ea411abf6df9199c55"
Oct 30 22:11:20.721198 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.721180    5087 scope.go:122] "RemoveContainer" containerID="ed425bb89ff342c4a1061e01a0f7e2011ab1194812c0b379a84bdba67dd54487"
Oct 30 22:11:20.731530 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.731507    5087 scope.go:122] "RemoveContainer" containerID="cf0513a665aab5adf830e5842a34964c8a2609604c7220ea411abf6df9199c55"
Oct 30 22:11:20.731809 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:11:20.731790    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf0513a665aab5adf830e5842a34964c8a2609604c7220ea411abf6df9199c55\": container with ID starting with cf0513a665aab5adf830e5842a34964c8a2609604c7220ea411abf6df9199c55 not found: ID does not exist" containerID="cf0513a665aab5adf830e5842a34964c8a2609604c7220ea411abf6df9199c55"
Oct 30 22:11:20.731888 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.731823    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf0513a665aab5adf830e5842a34964c8a2609604c7220ea411abf6df9199c55"} err="failed to get container status \"cf0513a665aab5adf830e5842a34964c8a2609604c7220ea411abf6df9199c55\": rpc error: code = NotFound desc = could not find container \"cf0513a665aab5adf830e5842a34964c8a2609604c7220ea411abf6df9199c55\": container with ID starting with cf0513a665aab5adf830e5842a34964c8a2609604c7220ea411abf6df9199c55 not found: ID does not exist"
Oct 30 22:11:20.731888 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.731848    5087 scope.go:122] "RemoveContainer" containerID="ed425bb89ff342c4a1061e01a0f7e2011ab1194812c0b379a84bdba67dd54487"
Oct 30 22:11:20.732099 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:11:20.732077    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed425bb89ff342c4a1061e01a0f7e2011ab1194812c0b379a84bdba67dd54487\": container with ID starting with ed425bb89ff342c4a1061e01a0f7e2011ab1194812c0b379a84bdba67dd54487 not found: ID does not exist" containerID="ed425bb89ff342c4a1061e01a0f7e2011ab1194812c0b379a84bdba67dd54487"
Oct 30 22:11:20.732137 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.732108    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed425bb89ff342c4a1061e01a0f7e2011ab1194812c0b379a84bdba67dd54487"} err="failed to get container status \"ed425bb89ff342c4a1061e01a0f7e2011ab1194812c0b379a84bdba67dd54487\": rpc error: code = NotFound desc = could not find container \"ed425bb89ff342c4a1061e01a0f7e2011ab1194812c0b379a84bdba67dd54487\": container with ID starting with ed425bb89ff342c4a1061e01a0f7e2011ab1194812c0b379a84bdba67dd54487 not found: ID does not exist"
Oct 30 22:11:20.873386 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.873320    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"]
Oct 30 22:11:20.908079 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:20.908049    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9c8c4d-openshift-default-dfc9cb655-fzv82"]
Oct 30 22:11:21.738368 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:21.737728    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c975ae55-4f25-494b-acd0-7458374253df" path="/var/lib/kubelet/pods/c975ae55-4f25-494b-acd0-7458374253df/volumes"
Oct 30 22:11:21.738368 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:21.738338    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"]
Oct 30 22:11:21.971168 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:21.971128    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"]
Oct 30 22:11:21.971403 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:21.971384    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5" podUID="9db8cae8-a410-4c25-bdec-cee65fb1a8ae" containerName="istio-proxy" containerID="cri-o://c8d1e942eeb3d42c6466daabdb66e4ffb8bc859c036ba5b9caf71d08ea040b3f" gracePeriod=30
Oct 30 22:11:23.877074 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:23.877028    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw" podUID="14ebb851-d1f0-43a8-8351-9fbe3e5a4e71" containerName="payload-processing" containerID="cri-o://e01ff47a8e18f1b2569c9ab88e80f25759536c33cea399cf3d8e7178fc6cdcf9" gracePeriod=25
Oct 30 22:11:23.882680 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:23.882651    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:11:23.882794 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:23.882716    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5" podUID="9db8cae8-a410-4c25-bdec-cee65fb1a8ae" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:11:23.986395 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:23.986333    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v" podUID="88a9407e-afe4-4e0b-af28-20226fb487c0" containerName="payload-pre-processing" containerID="cri-o://bf880dd59f6d51e81b9cf036e97aae707e3315cb42adf4b9002491dccd609583" gracePeriod=25
Oct 30 22:11:27.083326 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.083292    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6df3ba0e-7455bf6cc9-mcc6n"]
Oct 30 22:11:27.083904 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.083887    5087 cpu_manager.go:414] "containerMap: removing container" podUID="8cde2a2e-3765-4cd3-a1a8-ac979f272afc" containerName="revoke-keys"
Oct 30 22:11:27.083904 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.083903    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cde2a2e-3765-4cd3-a1a8-ac979f272afc" containerName="revoke-keys"
Oct 30 22:11:27.084026 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.083919    5087 cpu_manager.go:414] "containerMap: removing container" podUID="c975ae55-4f25-494b-acd0-7458374253df" containerName="istio-proxy"
Oct 30 22:11:27.084026 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.083924    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c975ae55-4f25-494b-acd0-7458374253df" containerName="istio-proxy"
Oct 30 22:11:27.084026 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.083941    5087 cpu_manager.go:414] "containerMap: removing container" podUID="a1269e40-c1fe-4166-ad83-6f07a8c95169" containerName="revoke-keys"
Oct 30 22:11:27.084026 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.083946    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1269e40-c1fe-4166-ad83-6f07a8c95169" containerName="revoke-keys"
Oct 30 22:11:27.084226 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.084040    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1269e40-c1fe-4166-ad83-6f07a8c95169" containerName="revoke-keys"
Oct 30 22:11:27.084226 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.084050    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="c975ae55-4f25-494b-acd0-7458374253df" containerName="istio-proxy"
Oct 30 22:11:27.084226 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.084061    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cde2a2e-3765-4cd3-a1a8-ac979f272afc" containerName="revoke-keys"
Oct 30 22:11:27.087424 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.087404    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6df3ba0e-7455bf6cc9-mcc6n"
Oct 30 22:11:27.206367 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.191696    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6df3ba0e\""
Oct 30 22:11:27.206367 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.191784    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6df3ba0e-dockercfg-jmqzn\""
Oct 30 22:11:27.206367 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.199435    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6df3ba0e-7455bf6cc9-mcc6n"]
Oct 30 22:11:27.247306 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.247274    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6df3ba0e-84d7646477-6qgzr"]
Oct 30 22:11:27.248197 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.248176    5087 cpu_manager.go:414] "containerMap: removing container" podUID="c975ae55-4f25-494b-acd0-7458374253df" containerName="istio-proxy"
Oct 30 22:11:27.248252 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.248200    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c975ae55-4f25-494b-acd0-7458374253df" containerName="istio-proxy"
Oct 30 22:11:27.248430 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.248389    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="c975ae55-4f25-494b-acd0-7458374253df" containerName="istio-proxy"
Oct 30 22:11:27.252547 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.252518    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6df3ba0e-84d7646477-6qgzr"
Oct 30 22:11:27.292452 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.292422    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6df3ba0e-84d7646477-6qgzr"]
Oct 30 22:11:27.299142 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.299111    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6df3ba0e-7455bf6cc9-mcc6n\" (UID: \"a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6df3ba0e-7455bf6cc9-mcc6n"
Oct 30 22:11:27.299297 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.299267    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89gc7\" (UniqueName: \"kubernetes.io/projected/a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a-kube-api-access-89gc7\") pod \"payload-pre-processing-e2e-mt-6df3ba0e-7455bf6cc9-mcc6n\" (UID: \"a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6df3ba0e-7455bf6cc9-mcc6n"
Oct 30 22:11:27.300254 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.300230    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66jd6\" (UniqueName: \"kubernetes.io/projected/8834311d-0639-467b-b6a1-4dddd34f83de-kube-api-access-66jd6\") pod \"payload-processing-e2e-mt-6df3ba0e-84d7646477-6qgzr\" (UID: \"8834311d-0639-467b-b6a1-4dddd34f83de\") " pod="openshift-ingress/payload-processing-e2e-mt-6df3ba0e-84d7646477-6qgzr"
Oct 30 22:11:27.300390 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.300279    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8834311d-0639-467b-b6a1-4dddd34f83de-plugins-config-volume\") pod \"payload-processing-e2e-mt-6df3ba0e-84d7646477-6qgzr\" (UID: \"8834311d-0639-467b-b6a1-4dddd34f83de\") " pod="openshift-ingress/payload-processing-e2e-mt-6df3ba0e-84d7646477-6qgzr"
Oct 30 22:11:27.333628 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.333598    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p"]
Oct 30 22:11:27.337839 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.337821    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p"
Oct 30 22:11:27.391521 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.391494    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:11:27.391521 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.391476    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:11:27.396835 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.396808    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p"]
Oct 30 22:11:27.401810 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.401781    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89gc7\" (UniqueName: \"kubernetes.io/projected/a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a-kube-api-access-89gc7\") pod \"payload-pre-processing-e2e-mt-6df3ba0e-7455bf6cc9-mcc6n\" (UID: \"a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6df3ba0e-7455bf6cc9-mcc6n"
Oct 30 22:11:27.401897 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.401866    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66jd6\" (UniqueName: \"kubernetes.io/projected/8834311d-0639-467b-b6a1-4dddd34f83de-kube-api-access-66jd6\") pod \"payload-processing-e2e-mt-6df3ba0e-84d7646477-6qgzr\" (UID: \"8834311d-0639-467b-b6a1-4dddd34f83de\") " pod="openshift-ingress/payload-processing-e2e-mt-6df3ba0e-84d7646477-6qgzr"
Oct 30 22:11:27.401985 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.401901    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8834311d-0639-467b-b6a1-4dddd34f83de-plugins-config-volume\") pod \"payload-processing-e2e-mt-6df3ba0e-84d7646477-6qgzr\" (UID: \"8834311d-0639-467b-b6a1-4dddd34f83de\") " pod="openshift-ingress/payload-processing-e2e-mt-6df3ba0e-84d7646477-6qgzr"
Oct 30 22:11:27.402030 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.401977    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw4hs\" (UniqueName: \"kubernetes.io/projected/52b45001-b134-41d3-ae1c-c30ad4e265d4-kube-api-access-cw4hs\") pod \"maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p\" (UID: \"52b45001-b134-41d3-ae1c-c30ad4e265d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p"
Oct 30 22:11:27.402030 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.402007    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52b45001-b134-41d3-ae1c-c30ad4e265d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p\" (UID: \"52b45001-b134-41d3-ae1c-c30ad4e265d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p"
Oct 30 22:11:27.402128 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.402080    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6df3ba0e-7455bf6cc9-mcc6n\" (UID: \"a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6df3ba0e-7455bf6cc9-mcc6n"
Oct 30 22:11:27.402128 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.402106    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52b45001-b134-41d3-ae1c-c30ad4e265d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p\" (UID: \"52b45001-b134-41d3-ae1c-c30ad4e265d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p"
Oct 30 22:11:27.403139 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.403117    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8834311d-0639-467b-b6a1-4dddd34f83de-plugins-config-volume\") pod \"payload-processing-e2e-mt-6df3ba0e-84d7646477-6qgzr\" (UID: \"8834311d-0639-467b-b6a1-4dddd34f83de\") " pod="openshift-ingress/payload-processing-e2e-mt-6df3ba0e-84d7646477-6qgzr"
Oct 30 22:11:27.403693 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.403673    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6df3ba0e-7455bf6cc9-mcc6n\" (UID: \"a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6df3ba0e-7455bf6cc9-mcc6n"
Oct 30 22:11:27.459014 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.458986    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66jd6\" (UniqueName: \"kubernetes.io/projected/8834311d-0639-467b-b6a1-4dddd34f83de-kube-api-access-66jd6\") pod \"payload-processing-e2e-mt-6df3ba0e-84d7646477-6qgzr\" (UID: \"8834311d-0639-467b-b6a1-4dddd34f83de\") " pod="openshift-ingress/payload-processing-e2e-mt-6df3ba0e-84d7646477-6qgzr"
Oct 30 22:11:27.459014 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.459005    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89gc7\" (UniqueName: \"kubernetes.io/projected/a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a-kube-api-access-89gc7\") pod \"payload-pre-processing-e2e-mt-6df3ba0e-7455bf6cc9-mcc6n\" (UID: \"a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6df3ba0e-7455bf6cc9-mcc6n"
Oct 30 22:11:27.503333 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.503259    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw4hs\" (UniqueName: \"kubernetes.io/projected/52b45001-b134-41d3-ae1c-c30ad4e265d4-kube-api-access-cw4hs\") pod \"maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p\" (UID: \"52b45001-b134-41d3-ae1c-c30ad4e265d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p"
Oct 30 22:11:27.503333 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.503297    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52b45001-b134-41d3-ae1c-c30ad4e265d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p\" (UID: \"52b45001-b134-41d3-ae1c-c30ad4e265d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p"
Oct 30 22:11:27.503557 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.503405    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52b45001-b134-41d3-ae1c-c30ad4e265d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p\" (UID: \"52b45001-b134-41d3-ae1c-c30ad4e265d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p"
Oct 30 22:11:27.504245 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.504172    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52b45001-b134-41d3-ae1c-c30ad4e265d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p\" (UID: \"52b45001-b134-41d3-ae1c-c30ad4e265d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p"
Oct 30 22:11:27.505851 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.505826    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52b45001-b134-41d3-ae1c-c30ad4e265d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p\" (UID: \"52b45001-b134-41d3-ae1c-c30ad4e265d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p"
Oct 30 22:11:27.560783 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.560753    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw4hs\" (UniqueName: \"kubernetes.io/projected/52b45001-b134-41d3-ae1c-c30ad4e265d4-kube-api-access-cw4hs\") pod \"maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p\" (UID: \"52b45001-b134-41d3-ae1c-c30ad4e265d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p"
Oct 30 22:11:27.568631 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.568570    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6df3ba0e-84d7646477-6qgzr"
Oct 30 22:11:27.659862 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.659834    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p"
Oct 30 22:11:27.672874 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.672650    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 22:11:27.700962 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.700810    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6df3ba0e-7455bf6cc9-mcc6n"
Oct 30 22:11:27.706031 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.705441    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-workload-socket\") pod \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") "
Oct 30 22:11:27.706031 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.705523    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istiod-ca-cert\") pod \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") "
Oct 30 22:11:27.706031 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.705557    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-kube-api-access-ctjwd\" (UniqueName: \"kubernetes.io/projected/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-kube-api-access-ctjwd\") pod \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") "
Oct 30 22:11:27.706031 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.705597    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istio-envoy\") pod \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") "
Oct 30 22:11:27.706031 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.705698    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istio-data\") pod \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") "
Oct 30 22:11:27.706031 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.705748    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-credential-socket\") pod \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") "
Oct 30 22:11:27.706031 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.705780    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istio-token\" (UniqueName: \"kubernetes.io/projected/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istio-token\") pod \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") "
Oct 30 22:11:27.706031 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.705899    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-workload-certs\") pod \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") "
Oct 30 22:11:27.706031 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.705950    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istiod-ca-cert" pod "9db8cae8-a410-4c25-bdec-cee65fb1a8ae" (UID: "9db8cae8-a410-4c25-bdec-cee65fb1a8ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:27.706569 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.706137    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-workload-socket" pod "9db8cae8-a410-4c25-bdec-cee65fb1a8ae" (UID: "9db8cae8-a410-4c25-bdec-cee65fb1a8ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:27.706569 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.706431    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istio-data" pod "9db8cae8-a410-4c25-bdec-cee65fb1a8ae" (UID: "9db8cae8-a410-4c25-bdec-cee65fb1a8ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:27.706823 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.705945    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istio-podinfo\") pod \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\" (UID: \"9db8cae8-a410-4c25-bdec-cee65fb1a8ae\") "
Oct 30 22:11:27.708068 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.707322    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:27.708068 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.707373    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:27.708068 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.707388    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:27.708829 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.708698    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-workload-certs" pod "9db8cae8-a410-4c25-bdec-cee65fb1a8ae" (UID: "9db8cae8-a410-4c25-bdec-cee65fb1a8ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:27.708829 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.708760    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-credential-socket" pod "9db8cae8-a410-4c25-bdec-cee65fb1a8ae" (UID: "9db8cae8-a410-4c25-bdec-cee65fb1a8ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:27.709368 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.709103    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-kube-api-access-ctjwd" pod "9db8cae8-a410-4c25-bdec-cee65fb1a8ae" (UID: "9db8cae8-a410-4c25-bdec-cee65fb1a8ae"). InnerVolumeSpecName "kube-api-access-ctjwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:27.709450 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.709408    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istio-envoy" pod "9db8cae8-a410-4c25-bdec-cee65fb1a8ae" (UID: "9db8cae8-a410-4c25-bdec-cee65fb1a8ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:27.710438 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.710404    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istio-token" pod "9db8cae8-a410-4c25-bdec-cee65fb1a8ae" (UID: "9db8cae8-a410-4c25-bdec-cee65fb1a8ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:27.710697 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.710670    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istio-podinfo" pod "9db8cae8-a410-4c25-bdec-cee65fb1a8ae" (UID: "9db8cae8-a410-4c25-bdec-cee65fb1a8ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:11:27.808453 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.808426    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:27.808453 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.808452    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:27.808668 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.808464    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctjwd\" (UniqueName: \"kubernetes.io/projected/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-kube-api-access-ctjwd\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:27.808668 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.808474    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:27.808668 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.808483    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:27.808668 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.808491    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db8cae8-a410-4c25-bdec-cee65fb1a8ae-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:27.813730 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.813701    5087 generic.go:356] "Generic (PLEG): container finished" podID="9db8cae8-a410-4c25-bdec-cee65fb1a8ae" containerID="c8d1e942eeb3d42c6466daabdb66e4ffb8bc859c036ba5b9caf71d08ea040b3f" exitCode=0
Oct 30 22:11:27.813859 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.813754    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"
Oct 30 22:11:27.813859 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.813829    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5" event={"ID":"9db8cae8-a410-4c25-bdec-cee65fb1a8ae","Type":"ContainerDied","Data":"c8d1e942eeb3d42c6466daabdb66e4ffb8bc859c036ba5b9caf71d08ea040b3f"}
Oct 30 22:11:27.813979 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.813868    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5" event={"ID":"9db8cae8-a410-4c25-bdec-cee65fb1a8ae","Type":"ContainerDied","Data":"91949efa101975e9eaef2bcaf421d5847138e0ba841e2aef3a5f91bb4daf2d45"}
Oct 30 22:11:27.813979 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.813897    5087 scope.go:122] "RemoveContainer" containerID="c8d1e942eeb3d42c6466daabdb66e4ffb8bc859c036ba5b9caf71d08ea040b3f"
Oct 30 22:11:27.830167 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.830125    5087 scope.go:122] "RemoveContainer" containerID="c8d1e942eeb3d42c6466daabdb66e4ffb8bc859c036ba5b9caf71d08ea040b3f"
Oct 30 22:11:27.830630 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:11:27.830602    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8d1e942eeb3d42c6466daabdb66e4ffb8bc859c036ba5b9caf71d08ea040b3f\": container with ID starting with c8d1e942eeb3d42c6466daabdb66e4ffb8bc859c036ba5b9caf71d08ea040b3f not found: ID does not exist" containerID="c8d1e942eeb3d42c6466daabdb66e4ffb8bc859c036ba5b9caf71d08ea040b3f"
Oct 30 22:11:27.830772 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.830643    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8d1e942eeb3d42c6466daabdb66e4ffb8bc859c036ba5b9caf71d08ea040b3f"} err="failed to get container status \"c8d1e942eeb3d42c6466daabdb66e4ffb8bc859c036ba5b9caf71d08ea040b3f\": rpc error: code = NotFound desc = could not find container \"c8d1e942eeb3d42c6466daabdb66e4ffb8bc859c036ba5b9caf71d08ea040b3f\": container with ID starting with c8d1e942eeb3d42c6466daabdb66e4ffb8bc859c036ba5b9caf71d08ea040b3f not found: ID does not exist"
Oct 30 22:11:27.851939 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.851761    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main" containerID="cri-o://b7624c27434712232a8e2c80556fcfb51de0bb56293f6d35b78a0784cd3766ef" gracePeriod=45
Oct 30 22:11:27.871614 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.871469    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main" containerID="cri-o://40f8dbd7ac6ec213cbf496c4c8949e4952f537c82625d058b7eac43d56bccc7f" gracePeriod=45
Oct 30 22:11:27.969723 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.969693    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6df3ba0e-84d7646477-6qgzr"]
Oct 30 22:11:27.976501 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:11:27.976475    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8834311d_0639_467b_b6a1_4dddd34f83de.slice/crio-4bdca920021dc61988f374e03b95bed136a9664a7ed9580f8bca2b06c29cf2c3 WatchSource:0}: Error finding container 4bdca920021dc61988f374e03b95bed136a9664a7ed9580f8bca2b06c29cf2c3: Status 404 returned error can't find the container with id 4bdca920021dc61988f374e03b95bed136a9664a7ed9580f8bca2b06c29cf2c3
Oct 30 22:11:27.990935 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.990911    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"]
Oct 30 22:11:27.998676 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:27.998650    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-vqsb5"]
Oct 30 22:11:28.089917 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.089740    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p"]
Oct 30 22:11:28.141473 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.141444    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6df3ba0e-7455bf6cc9-mcc6n"]
Oct 30 22:11:28.203098 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.203073    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 22:11:28.230403 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.230328    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 22:11:28.326561 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.326470    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8f983a-9982-4ec1-82d6-a76c70838a8c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd8f983a-9982-4ec1-82d6-a76c70838a8c-dshm\") pod \"fd8f983a-9982-4ec1-82d6-a76c70838a8c\" (UID: \"fd8f983a-9982-4ec1-82d6-a76c70838a8c\") "
Oct 30 22:11:28.326561 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.326522    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd8f983a-9982-4ec1-82d6-a76c70838a8c-kube-api-access-nlp69\" (UniqueName: \"kubernetes.io/projected/fd8f983a-9982-4ec1-82d6-a76c70838a8c-kube-api-access-nlp69\") pod \"fd8f983a-9982-4ec1-82d6-a76c70838a8c\" (UID: \"fd8f983a-9982-4ec1-82d6-a76c70838a8c\") "
Oct 30 22:11:28.326802 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.326580    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/777cc409-27c5-48c2-9f31-8f64bc4744dc-tls-certs\" (UniqueName: \"kubernetes.io/secret/777cc409-27c5-48c2-9f31-8f64bc4744dc-tls-certs\") pod \"777cc409-27c5-48c2-9f31-8f64bc4744dc\" (UID: \"777cc409-27c5-48c2-9f31-8f64bc4744dc\") "
Oct 30 22:11:28.326802 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.326617    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8f983a-9982-4ec1-82d6-a76c70838a8c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd8f983a-9982-4ec1-82d6-a76c70838a8c-tmp-dir\") pod \"fd8f983a-9982-4ec1-82d6-a76c70838a8c\" (UID: \"fd8f983a-9982-4ec1-82d6-a76c70838a8c\") "
Oct 30 22:11:28.326802 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.326662    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8f983a-9982-4ec1-82d6-a76c70838a8c-home\" (UniqueName: \"kubernetes.io/empty-dir/fd8f983a-9982-4ec1-82d6-a76c70838a8c-home\") pod \"fd8f983a-9982-4ec1-82d6-a76c70838a8c\" (UID: \"fd8f983a-9982-4ec1-82d6-a76c70838a8c\") "
Oct 30 22:11:28.326802 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.326705    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/777cc409-27c5-48c2-9f31-8f64bc4744dc-kube-api-access-t89cb\" (UniqueName: \"kubernetes.io/projected/777cc409-27c5-48c2-9f31-8f64bc4744dc-kube-api-access-t89cb\") pod \"777cc409-27c5-48c2-9f31-8f64bc4744dc\" (UID: \"777cc409-27c5-48c2-9f31-8f64bc4744dc\") "
Oct 30 22:11:28.326802 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.326737    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/777cc409-27c5-48c2-9f31-8f64bc4744dc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/777cc409-27c5-48c2-9f31-8f64bc4744dc-dshm\") pod \"777cc409-27c5-48c2-9f31-8f64bc4744dc\" (UID: \"777cc409-27c5-48c2-9f31-8f64bc4744dc\") "
Oct 30 22:11:28.326802 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.326780    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/777cc409-27c5-48c2-9f31-8f64bc4744dc-home\" (UniqueName: \"kubernetes.io/empty-dir/777cc409-27c5-48c2-9f31-8f64bc4744dc-home\") pod \"777cc409-27c5-48c2-9f31-8f64bc4744dc\" (UID: \"777cc409-27c5-48c2-9f31-8f64bc4744dc\") "
Oct 30 22:11:28.327089 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.326850    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8f983a-9982-4ec1-82d6-a76c70838a8c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd8f983a-9982-4ec1-82d6-a76c70838a8c-model-cache\") pod \"fd8f983a-9982-4ec1-82d6-a76c70838a8c\" (UID: \"fd8f983a-9982-4ec1-82d6-a76c70838a8c\") "
Oct 30 22:11:28.327089 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.326904    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd8f983a-9982-4ec1-82d6-a76c70838a8c-tls-certs\" (UniqueName: \"kubernetes.io/secret/fd8f983a-9982-4ec1-82d6-a76c70838a8c-tls-certs\") pod \"fd8f983a-9982-4ec1-82d6-a76c70838a8c\" (UID: \"fd8f983a-9982-4ec1-82d6-a76c70838a8c\") "
Oct 30 22:11:28.327089 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.326951    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/777cc409-27c5-48c2-9f31-8f64bc4744dc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/777cc409-27c5-48c2-9f31-8f64bc4744dc-model-cache\") pod \"777cc409-27c5-48c2-9f31-8f64bc4744dc\" (UID: \"777cc409-27c5-48c2-9f31-8f64bc4744dc\") "
Oct 30 22:11:28.327089 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.327003    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/777cc409-27c5-48c2-9f31-8f64bc4744dc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/777cc409-27c5-48c2-9f31-8f64bc4744dc-tmp-dir\") pod \"777cc409-27c5-48c2-9f31-8f64bc4744dc\" (UID: \"777cc409-27c5-48c2-9f31-8f64bc4744dc\") "
Oct 30 22:11:28.327766 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.327728    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/777cc409-27c5-48c2-9f31-8f64bc4744dc-tmp-dir" pod "777cc409-27c5-48c2-9f31-8f64bc4744dc" (UID: "777cc409-27c5-48c2-9f31-8f64bc4744dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:28.327893 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.327738    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8f983a-9982-4ec1-82d6-a76c70838a8c-model-cache" pod "fd8f983a-9982-4ec1-82d6-a76c70838a8c" (UID: "fd8f983a-9982-4ec1-82d6-a76c70838a8c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:28.327977 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.327953    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/777cc409-27c5-48c2-9f31-8f64bc4744dc-home" pod "777cc409-27c5-48c2-9f31-8f64bc4744dc" (UID: "777cc409-27c5-48c2-9f31-8f64bc4744dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:28.328325 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.328241    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/777cc409-27c5-48c2-9f31-8f64bc4744dc-model-cache" pod "777cc409-27c5-48c2-9f31-8f64bc4744dc" (UID: "777cc409-27c5-48c2-9f31-8f64bc4744dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:28.328325 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.328313    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8f983a-9982-4ec1-82d6-a76c70838a8c-home" pod "fd8f983a-9982-4ec1-82d6-a76c70838a8c" (UID: "fd8f983a-9982-4ec1-82d6-a76c70838a8c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:28.328769 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.328465    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8f983a-9982-4ec1-82d6-a76c70838a8c-tmp-dir" pod "fd8f983a-9982-4ec1-82d6-a76c70838a8c" (UID: "fd8f983a-9982-4ec1-82d6-a76c70838a8c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:28.342160 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.342130    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8f983a-9982-4ec1-82d6-a76c70838a8c-dshm" pod "fd8f983a-9982-4ec1-82d6-a76c70838a8c" (UID: "fd8f983a-9982-4ec1-82d6-a76c70838a8c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:28.343204 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.343163    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777cc409-27c5-48c2-9f31-8f64bc4744dc-kube-api-access-t89cb" pod "777cc409-27c5-48c2-9f31-8f64bc4744dc" (UID: "777cc409-27c5-48c2-9f31-8f64bc4744dc"). InnerVolumeSpecName "kube-api-access-t89cb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:28.345740 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.345709    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd8f983a-9982-4ec1-82d6-a76c70838a8c-tls-certs" pod "fd8f983a-9982-4ec1-82d6-a76c70838a8c" (UID: "fd8f983a-9982-4ec1-82d6-a76c70838a8c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:28.346320 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.346233    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/777cc409-27c5-48c2-9f31-8f64bc4744dc-tls-certs" pod "777cc409-27c5-48c2-9f31-8f64bc4744dc" (UID: "777cc409-27c5-48c2-9f31-8f64bc4744dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:28.346320 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.346301    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/777cc409-27c5-48c2-9f31-8f64bc4744dc-dshm" pod "777cc409-27c5-48c2-9f31-8f64bc4744dc" (UID: "777cc409-27c5-48c2-9f31-8f64bc4744dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:28.353135 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.353103    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8f983a-9982-4ec1-82d6-a76c70838a8c-kube-api-access-nlp69" pod "fd8f983a-9982-4ec1-82d6-a76c70838a8c" (UID: "fd8f983a-9982-4ec1-82d6-a76c70838a8c"). InnerVolumeSpecName "kube-api-access-nlp69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:28.428716 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.428685    5087 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd8f983a-9982-4ec1-82d6-a76c70838a8c-model-cache\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:28.428716 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.428712    5087 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd8f983a-9982-4ec1-82d6-a76c70838a8c-tls-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:28.428716 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.428721    5087 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/777cc409-27c5-48c2-9f31-8f64bc4744dc-model-cache\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:28.428936 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.428730    5087 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/777cc409-27c5-48c2-9f31-8f64bc4744dc-tmp-dir\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:28.428936 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.428740    5087 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd8f983a-9982-4ec1-82d6-a76c70838a8c-dshm\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:28.428936 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.428749    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlp69\" (UniqueName: \"kubernetes.io/projected/fd8f983a-9982-4ec1-82d6-a76c70838a8c-kube-api-access-nlp69\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:28.428936 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.428757    5087 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/777cc409-27c5-48c2-9f31-8f64bc4744dc-tls-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:28.428936 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.428764    5087 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd8f983a-9982-4ec1-82d6-a76c70838a8c-tmp-dir\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:28.428936 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.428771    5087 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd8f983a-9982-4ec1-82d6-a76c70838a8c-home\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:28.428936 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.428779    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t89cb\" (UniqueName: \"kubernetes.io/projected/777cc409-27c5-48c2-9f31-8f64bc4744dc-kube-api-access-t89cb\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:28.428936 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.428786    5087 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/777cc409-27c5-48c2-9f31-8f64bc4744dc-dshm\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:28.428936 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.428795    5087 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/777cc409-27c5-48c2-9f31-8f64bc4744dc-home\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:28.822434 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.822397    5087 generic.go:356] "Generic (PLEG): container finished" podID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerID="b7624c27434712232a8e2c80556fcfb51de0bb56293f6d35b78a0784cd3766ef" exitCode=0
Oct 30 22:11:28.822634 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.822449    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"
Oct 30 22:11:28.822634 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.822493    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" event={"ID":"fd8f983a-9982-4ec1-82d6-a76c70838a8c","Type":"ContainerDied","Data":"b7624c27434712232a8e2c80556fcfb51de0bb56293f6d35b78a0784cd3766ef"}
Oct 30 22:11:28.822634 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.822545    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch" event={"ID":"fd8f983a-9982-4ec1-82d6-a76c70838a8c","Type":"ContainerDied","Data":"6517ac240fd75f2cdf017dc919c77edea5313c9fc7e237586865f7f796c9a66b"}
Oct 30 22:11:28.822634 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.822568    5087 scope.go:122] "RemoveContainer" containerID="b7624c27434712232a8e2c80556fcfb51de0bb56293f6d35b78a0784cd3766ef"
Oct 30 22:11:28.824399 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.824335    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6df3ba0e-84d7646477-6qgzr" event={"ID":"8834311d-0639-467b-b6a1-4dddd34f83de","Type":"ContainerStarted","Data":"06cbaee83690f219920c6c7ca411592fd764b5b124dc4f91457f7a779336cce6"}
Oct 30 22:11:28.824526 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.824423    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6df3ba0e-84d7646477-6qgzr" event={"ID":"8834311d-0639-467b-b6a1-4dddd34f83de","Type":"ContainerStarted","Data":"4bdca920021dc61988f374e03b95bed136a9664a7ed9580f8bca2b06c29cf2c3"}
Oct 30 22:11:28.824593 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.824540    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6df3ba0e-84d7646477-6qgzr"
Oct 30 22:11:28.826073 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.826050    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p" event={"ID":"52b45001-b134-41d3-ae1c-c30ad4e265d4","Type":"ContainerStarted","Data":"f31f000eb28fa6f8c76a10e4b8cc57c12a744ae8e5d717ec5fb1490301846e12"}
Oct 30 22:11:28.826205 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.826088    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p" event={"ID":"52b45001-b134-41d3-ae1c-c30ad4e265d4","Type":"ContainerStarted","Data":"c9eba0f7ad39ebca4752581ccde249b2d0d27efd5f4c246c3a45d1a946afb8eb"}
Oct 30 22:11:28.828641 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.828621    5087 generic.go:356] "Generic (PLEG): container finished" podID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerID="40f8dbd7ac6ec213cbf496c4c8949e4952f537c82625d058b7eac43d56bccc7f" exitCode=0
Oct 30 22:11:28.828749 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.828670    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"
Oct 30 22:11:28.828749 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.828700    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" event={"ID":"777cc409-27c5-48c2-9f31-8f64bc4744dc","Type":"ContainerDied","Data":"40f8dbd7ac6ec213cbf496c4c8949e4952f537c82625d058b7eac43d56bccc7f"}
Oct 30 22:11:28.828749 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.828730    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt" event={"ID":"777cc409-27c5-48c2-9f31-8f64bc4744dc","Type":"ContainerDied","Data":"882eb7d446567f93857b33e9c49177d4d45a171efbd57bdb4f4004484939719f"}
Oct 30 22:11:28.830364 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.830325    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6df3ba0e-7455bf6cc9-mcc6n" event={"ID":"a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a","Type":"ContainerStarted","Data":"eb91986c766b9fa5fb0d1c8b444d252471029e8581d5f481c35b0e9d8ac63470"}
Oct 30 22:11:28.830447 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.830411    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6df3ba0e-7455bf6cc9-mcc6n"
Oct 30 22:11:28.830447 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.830434    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6df3ba0e-7455bf6cc9-mcc6n" event={"ID":"a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a","Type":"ContainerStarted","Data":"108841a0bb45cedf2db7acef24c4964bbcb57faf80005ba177ce82732b6726eb"}
Oct 30 22:11:28.837825 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.837802    5087 scope.go:122] "RemoveContainer" containerID="b7624c27434712232a8e2c80556fcfb51de0bb56293f6d35b78a0784cd3766ef"
Oct 30 22:11:28.838183 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:11:28.838149    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7624c27434712232a8e2c80556fcfb51de0bb56293f6d35b78a0784cd3766ef\": container with ID starting with b7624c27434712232a8e2c80556fcfb51de0bb56293f6d35b78a0784cd3766ef not found: ID does not exist" containerID="b7624c27434712232a8e2c80556fcfb51de0bb56293f6d35b78a0784cd3766ef"
Oct 30 22:11:28.838259 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.838184    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7624c27434712232a8e2c80556fcfb51de0bb56293f6d35b78a0784cd3766ef"} err="failed to get container status \"b7624c27434712232a8e2c80556fcfb51de0bb56293f6d35b78a0784cd3766ef\": rpc error: code = NotFound desc = could not find container \"b7624c27434712232a8e2c80556fcfb51de0bb56293f6d35b78a0784cd3766ef\": container with ID starting with b7624c27434712232a8e2c80556fcfb51de0bb56293f6d35b78a0784cd3766ef not found: ID does not exist"
Oct 30 22:11:28.838259 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.838209    5087 scope.go:122] "RemoveContainer" containerID="40f8dbd7ac6ec213cbf496c4c8949e4952f537c82625d058b7eac43d56bccc7f"
Oct 30 22:11:28.849926 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.849902    5087 scope.go:122] "RemoveContainer" containerID="40f8dbd7ac6ec213cbf496c4c8949e4952f537c82625d058b7eac43d56bccc7f"
Oct 30 22:11:28.850211 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:11:28.850189    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40f8dbd7ac6ec213cbf496c4c8949e4952f537c82625d058b7eac43d56bccc7f\": container with ID starting with 40f8dbd7ac6ec213cbf496c4c8949e4952f537c82625d058b7eac43d56bccc7f not found: ID does not exist" containerID="40f8dbd7ac6ec213cbf496c4c8949e4952f537c82625d058b7eac43d56bccc7f"
Oct 30 22:11:28.850283 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.850218    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40f8dbd7ac6ec213cbf496c4c8949e4952f537c82625d058b7eac43d56bccc7f"} err="failed to get container status \"40f8dbd7ac6ec213cbf496c4c8949e4952f537c82625d058b7eac43d56bccc7f\": rpc error: code = NotFound desc = could not find container \"40f8dbd7ac6ec213cbf496c4c8949e4952f537c82625d058b7eac43d56bccc7f\": container with ID starting with 40f8dbd7ac6ec213cbf496c4c8949e4952f537c82625d058b7eac43d56bccc7f not found: ID does not exist"
Oct 30 22:11:28.986669 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.986634    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"]
Oct 30 22:11:28.992187 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:28.992165    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-dda9f5/distinct-w0-dda9f5-kserve-b9946d498-tfpgt"]
Oct 30 22:11:29.016911 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:29.016884    5087 scope.go:122] "RemoveContainer" containerID="8c1484516cc037d23ee71b072f8dde098e9058bbe996ba0236d6b7f1b9a51d34"
Oct 30 22:11:29.027047 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:29.027025    5087 scope.go:122] "RemoveContainer" containerID="8c0e53a81e7fe393befbe0df60af66331c8288d0eca4b93deb949306d4da6b2d"
Oct 30 22:11:29.101070 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:29.101006    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6df3ba0e-7455bf6cc9-mcc6n" podStartSLOduration=2.100988264 podStartE2EDuration="2.100988264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:29.100384366 +0000 UTC m=+2283.574888349" watchObservedRunningTime="2026-10-30 22:11:29.100988264 +0000 UTC m=+2283.575492244"
Oct 30 22:11:29.329211 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:29.329109    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p" podStartSLOduration=2.329093707 podStartE2EDuration="2.329093707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:29.327386425 +0000 UTC m=+2283.801890429" watchObservedRunningTime="2026-10-30 22:11:29.329093707 +0000 UTC m=+2283.803597725"
Oct 30 22:11:29.329712 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:29.329684    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6df3ba0e-84d7646477-6qgzr" podStartSLOduration=2.3296760020000002 podStartE2EDuration="2.329676002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:29.214725451 +0000 UTC m=+2283.689229431" watchObservedRunningTime="2026-10-30 22:11:29.329676002 +0000 UTC m=+2283.804179975"
Oct 30 22:11:29.439629 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:29.439595    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"]
Oct 30 22:11:29.442525 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:29.442505    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-dda9f5/distinct-2-w0-dda9f5-kserve-6d65f89b7c-p2rch"]
Oct 30 22:11:29.710013 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:29.709942    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" path="/var/lib/kubelet/pods/777cc409-27c5-48c2-9f31-8f64bc4744dc/volumes"
Oct 30 22:11:29.710408 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:29.710394    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9db8cae8-a410-4c25-bdec-cee65fb1a8ae" path="/var/lib/kubelet/pods/9db8cae8-a410-4c25-bdec-cee65fb1a8ae/volumes"
Oct 30 22:11:29.710787 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:29.710775    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" path="/var/lib/kubelet/pods/fd8f983a-9982-4ec1-82d6-a76c70838a8c/volumes"
Oct 30 22:11:29.839038 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:29.838995    5087 generic.go:356] "Generic (PLEG): container finished" podID="52b45001-b134-41d3-ae1c-c30ad4e265d4" containerID="f31f000eb28fa6f8c76a10e4b8cc57c12a744ae8e5d717ec5fb1490301846e12" exitCode=0
Oct 30 22:11:29.839203 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:29.839089    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p" event={"ID":"52b45001-b134-41d3-ae1c-c30ad4e265d4","Type":"ContainerDied","Data":"f31f000eb28fa6f8c76a10e4b8cc57c12a744ae8e5d717ec5fb1490301846e12"}
Oct 30 22:11:31.022617 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:31.022595    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p"
Oct 30 22:11:31.159581 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:31.159482    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52b45001-b134-41d3-ae1c-c30ad4e265d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52b45001-b134-41d3-ae1c-c30ad4e265d4-maas-api-xks-ca\") pod \"52b45001-b134-41d3-ae1c-c30ad4e265d4\" (UID: \"52b45001-b134-41d3-ae1c-c30ad4e265d4\") "
Oct 30 22:11:31.159757 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:31.159655    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52b45001-b134-41d3-ae1c-c30ad4e265d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52b45001-b134-41d3-ae1c-c30ad4e265d4-maas-api-openshift-ca\") pod \"52b45001-b134-41d3-ae1c-c30ad4e265d4\" (UID: \"52b45001-b134-41d3-ae1c-c30ad4e265d4\") "
Oct 30 22:11:31.159757 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:31.159705    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52b45001-b134-41d3-ae1c-c30ad4e265d4-kube-api-access-cw4hs\" (UniqueName: \"kubernetes.io/projected/52b45001-b134-41d3-ae1c-c30ad4e265d4-kube-api-access-cw4hs\") pod \"52b45001-b134-41d3-ae1c-c30ad4e265d4\" (UID: \"52b45001-b134-41d3-ae1c-c30ad4e265d4\") "
Oct 30 22:11:31.160713 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:31.160687    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52b45001-b134-41d3-ae1c-c30ad4e265d4-maas-api-openshift-ca" pod "52b45001-b134-41d3-ae1c-c30ad4e265d4" (UID: "52b45001-b134-41d3-ae1c-c30ad4e265d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:31.162687 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:31.162662    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52b45001-b134-41d3-ae1c-c30ad4e265d4-maas-api-xks-ca" pod "52b45001-b134-41d3-ae1c-c30ad4e265d4" (UID: "52b45001-b134-41d3-ae1c-c30ad4e265d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:31.162933 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:31.162900    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52b45001-b134-41d3-ae1c-c30ad4e265d4-kube-api-access-cw4hs" pod "52b45001-b134-41d3-ae1c-c30ad4e265d4" (UID: "52b45001-b134-41d3-ae1c-c30ad4e265d4"). InnerVolumeSpecName "kube-api-access-cw4hs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:31.261011 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:31.260982    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52b45001-b134-41d3-ae1c-c30ad4e265d4-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:31.261011 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:31.261009    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52b45001-b134-41d3-ae1c-c30ad4e265d4-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:31.261212 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:31.261020    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw4hs\" (UniqueName: \"kubernetes.io/projected/52b45001-b134-41d3-ae1c-c30ad4e265d4-kube-api-access-cw4hs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:31.856522 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:31.856485    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p"
Oct 30 22:11:31.856522 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:31.856511    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p" event={"ID":"52b45001-b134-41d3-ae1c-c30ad4e265d4","Type":"ContainerDied","Data":"c9eba0f7ad39ebca4752581ccde249b2d0d27efd5f4c246c3a45d1a946afb8eb"}
Oct 30 22:11:31.856739 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:31.856566    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9eba0f7ad39ebca4752581ccde249b2d0d27efd5f4c246c3a45d1a946afb8eb"
Oct 30 22:11:32.550890 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:32.550858    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt"]
Oct 30 22:11:32.551340 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:32.551056    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt" podUID="cae946a0-125e-4832-96eb-8b824f689a88" containerName="maas-api" containerID="cri-o://2eb96bdd43dcbf0bf9cc88348babd0a34803f9262386e065df30df80637bbbdb" gracePeriod=30
Oct 30 22:11:32.881044 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:32.881014    5087 generic.go:356] "Generic (PLEG): container finished" podID="cae946a0-125e-4832-96eb-8b824f689a88" containerID="2eb96bdd43dcbf0bf9cc88348babd0a34803f9262386e065df30df80637bbbdb" exitCode=0
Oct 30 22:11:32.881183 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:32.881052    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt" event={"ID":"cae946a0-125e-4832-96eb-8b824f689a88","Type":"ContainerDied","Data":"2eb96bdd43dcbf0bf9cc88348babd0a34803f9262386e065df30df80637bbbdb"}
Oct 30 22:11:32.881183 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:32.881121    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt" event={"ID":"cae946a0-125e-4832-96eb-8b824f689a88","Type":"ContainerDied","Data":"abc4555092f92ff51b0a907a8daadeab113c1f1d0ab15c2d0c2361bd04e3e397"}
Oct 30 22:11:32.881183 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:32.881144    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abc4555092f92ff51b0a907a8daadeab113c1f1d0ab15c2d0c2361bd04e3e397"
Oct 30 22:11:32.890841 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:32.890815    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt"
Oct 30 22:11:32.979182 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:32.979136    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cae946a0-125e-4832-96eb-8b824f689a88-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae946a0-125e-4832-96eb-8b824f689a88-maas-api-tls\") pod \"cae946a0-125e-4832-96eb-8b824f689a88\" (UID: \"cae946a0-125e-4832-96eb-8b824f689a88\") "
Oct 30 22:11:32.979449 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:32.979259    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cae946a0-125e-4832-96eb-8b824f689a88-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae946a0-125e-4832-96eb-8b824f689a88-openshift-service-ca\") pod \"cae946a0-125e-4832-96eb-8b824f689a88\" (UID: \"cae946a0-125e-4832-96eb-8b824f689a88\") "
Oct 30 22:11:32.979449 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:32.979391    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cae946a0-125e-4832-96eb-8b824f689a88-kube-api-access-pvp9w\" (UniqueName: \"kubernetes.io/projected/cae946a0-125e-4832-96eb-8b824f689a88-kube-api-access-pvp9w\") pod \"cae946a0-125e-4832-96eb-8b824f689a88\" (UID: \"cae946a0-125e-4832-96eb-8b824f689a88\") "
Oct 30 22:11:32.979671 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:32.979459    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cae946a0-125e-4832-96eb-8b824f689a88-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae946a0-125e-4832-96eb-8b824f689a88-metrics-certs\") pod \"cae946a0-125e-4832-96eb-8b824f689a88\" (UID: \"cae946a0-125e-4832-96eb-8b824f689a88\") "
Oct 30 22:11:32.984650 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:32.982539    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cae946a0-125e-4832-96eb-8b824f689a88-openshift-service-ca" pod "cae946a0-125e-4832-96eb-8b824f689a88" (UID: "cae946a0-125e-4832-96eb-8b824f689a88"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:32.991598 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:32.991566    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cae946a0-125e-4832-96eb-8b824f689a88-kube-api-access-pvp9w" pod "cae946a0-125e-4832-96eb-8b824f689a88" (UID: "cae946a0-125e-4832-96eb-8b824f689a88"). InnerVolumeSpecName "kube-api-access-pvp9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:32.993795 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:32.993766    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cae946a0-125e-4832-96eb-8b824f689a88-metrics-certs" pod "cae946a0-125e-4832-96eb-8b824f689a88" (UID: "cae946a0-125e-4832-96eb-8b824f689a88"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:32.994277 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:32.994242    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cae946a0-125e-4832-96eb-8b824f689a88-maas-api-tls" pod "cae946a0-125e-4832-96eb-8b824f689a88" (UID: "cae946a0-125e-4832-96eb-8b824f689a88"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:33.081192 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:33.081160    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvp9w\" (UniqueName: \"kubernetes.io/projected/cae946a0-125e-4832-96eb-8b824f689a88-kube-api-access-pvp9w\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:33.081192 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:33.081188    5087 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cae946a0-125e-4832-96eb-8b824f689a88-metrics-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:33.081192 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:33.081200    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cae946a0-125e-4832-96eb-8b824f689a88-maas-api-tls\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:33.081555 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:33.081209    5087 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae946a0-125e-4832-96eb-8b824f689a88-openshift-service-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:33.330254 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:33.330214    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw" podUID="14ebb851-d1f0-43a8-8351-9fbe3e5a4e71" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:11:33.330254 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:33.330239    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v" podUID="88a9407e-afe4-4e0b-af28-20226fb487c0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:11:33.889000 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:33.888970    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt"
Oct 30 22:11:34.059893 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.059861    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt"]
Oct 30 22:11:34.108703 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.108677    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-1db768-7567866f5-zkrxt"]
Oct 30 22:11:34.117672 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.117645    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9"]
Oct 30 22:11:34.118173 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.118161    5087 cpu_manager.go:414] "containerMap: removing container" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main"
Oct 30 22:11:34.118213 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.118175    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main"
Oct 30 22:11:34.118213 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.118189    5087 cpu_manager.go:414] "containerMap: removing container" podUID="52b45001-b134-41d3-ae1c-c30ad4e265d4" containerName="revoke-keys"
Oct 30 22:11:34.118213 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.118194    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52b45001-b134-41d3-ae1c-c30ad4e265d4" containerName="revoke-keys"
Oct 30 22:11:34.118213 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.118204    5087 cpu_manager.go:414] "containerMap: removing container" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main"
Oct 30 22:11:34.118213 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.118208    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main"
Oct 30 22:11:34.118383 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.118223    5087 cpu_manager.go:414] "containerMap: removing container" podUID="cae946a0-125e-4832-96eb-8b824f689a88" containerName="maas-api"
Oct 30 22:11:34.118383 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.118228    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cae946a0-125e-4832-96eb-8b824f689a88" containerName="maas-api"
Oct 30 22:11:34.118383 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.118257    5087 cpu_manager.go:414] "containerMap: removing container" podUID="9db8cae8-a410-4c25-bdec-cee65fb1a8ae" containerName="istio-proxy"
Oct 30 22:11:34.118383 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.118262    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9db8cae8-a410-4c25-bdec-cee65fb1a8ae" containerName="istio-proxy"
Oct 30 22:11:34.118383 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.118334    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="777cc409-27c5-48c2-9f31-8f64bc4744dc" containerName="main"
Oct 30 22:11:34.118383 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.118358    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="cae946a0-125e-4832-96eb-8b824f689a88" containerName="maas-api"
Oct 30 22:11:34.118383 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.118369    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="9db8cae8-a410-4c25-bdec-cee65fb1a8ae" containerName="istio-proxy"
Oct 30 22:11:34.118383 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.118377    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd8f983a-9982-4ec1-82d6-a76c70838a8c" containerName="main"
Oct 30 22:11:34.118383 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.118383    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="52b45001-b134-41d3-ae1c-c30ad4e265d4" containerName="revoke-keys"
Oct 30 22:11:34.121577 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.121562    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9"
Oct 30 22:11:34.175123 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.175028    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-9da607-serving-cert\""
Oct 30 22:11:34.178924 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.178900    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9"]
Oct 30 22:11:34.193768 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.193740    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a031d03-d200-4477-9a61-8385a063abf1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9\" (UID: \"8a031d03-d200-4477-9a61-8385a063abf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9"
Oct 30 22:11:34.193901 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.193784    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a031d03-d200-4477-9a61-8385a063abf1-maas-api-tls\") pod \"maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9\" (UID: \"8a031d03-d200-4477-9a61-8385a063abf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9"
Oct 30 22:11:34.193956 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.193916    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfxzm\" (UniqueName: \"kubernetes.io/projected/8a031d03-d200-4477-9a61-8385a063abf1-kube-api-access-lfxzm\") pod \"maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9\" (UID: \"8a031d03-d200-4477-9a61-8385a063abf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9"
Oct 30 22:11:34.194077 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.194064    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a031d03-d200-4477-9a61-8385a063abf1-metrics-certs\") pod \"maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9\" (UID: \"8a031d03-d200-4477-9a61-8385a063abf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9"
Oct 30 22:11:34.295386 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.295318    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfxzm\" (UniqueName: \"kubernetes.io/projected/8a031d03-d200-4477-9a61-8385a063abf1-kube-api-access-lfxzm\") pod \"maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9\" (UID: \"8a031d03-d200-4477-9a61-8385a063abf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9"
Oct 30 22:11:34.295584 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.295484    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a031d03-d200-4477-9a61-8385a063abf1-metrics-certs\") pod \"maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9\" (UID: \"8a031d03-d200-4477-9a61-8385a063abf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9"
Oct 30 22:11:34.295584 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.295526    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a031d03-d200-4477-9a61-8385a063abf1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9\" (UID: \"8a031d03-d200-4477-9a61-8385a063abf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9"
Oct 30 22:11:34.295584 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.295552    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a031d03-d200-4477-9a61-8385a063abf1-maas-api-tls\") pod \"maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9\" (UID: \"8a031d03-d200-4477-9a61-8385a063abf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9"
Oct 30 22:11:34.295736 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:11:34.295690    5087 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-serving-cert: secret "maas-api-e2e-inf-a-9da607-serving-cert" not found
Oct 30 22:11:34.295805 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:11:34.295746    5087 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a031d03-d200-4477-9a61-8385a063abf1-maas-api-tls podName:8a031d03-d200-4477-9a61-8385a063abf1 nodeName:}" failed. No retries permitted until 2026-10-30 22:11:34.795727424 +0000 UTC m=+2289.270231381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8a031d03-d200-4477-9a61-8385a063abf1-maas-api-tls") pod "maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9" (UID: "8a031d03-d200-4477-9a61-8385a063abf1") : secret "maas-api-e2e-inf-a-9da607-serving-cert" not found
Oct 30 22:11:34.296213 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.296193    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a031d03-d200-4477-9a61-8385a063abf1-openshift-service-ca\") pod \"maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9\" (UID: \"8a031d03-d200-4477-9a61-8385a063abf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9"
Oct 30 22:11:34.297989 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.297969    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a031d03-d200-4477-9a61-8385a063abf1-metrics-certs\") pod \"maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9\" (UID: \"8a031d03-d200-4477-9a61-8385a063abf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9"
Oct 30 22:11:34.352670 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.352566    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfxzm\" (UniqueName: \"kubernetes.io/projected/8a031d03-d200-4477-9a61-8385a063abf1-kube-api-access-lfxzm\") pod \"maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9\" (UID: \"8a031d03-d200-4477-9a61-8385a063abf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9"
Oct 30 22:11:34.764910 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.764876    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1db768-5z5fd"]
Oct 30 22:11:34.769403 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.769366    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-1db768-5z5fd"]
Oct 30 22:11:34.800880 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.800848    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a031d03-d200-4477-9a61-8385a063abf1-maas-api-tls\") pod \"maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9\" (UID: \"8a031d03-d200-4477-9a61-8385a063abf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9"
Oct 30 22:11:34.803585 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:34.803564    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a031d03-d200-4477-9a61-8385a063abf1-maas-api-tls\") pod \"maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9\" (UID: \"8a031d03-d200-4477-9a61-8385a063abf1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9"
Oct 30 22:11:35.035165 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:35.035074    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9"
Oct 30 22:11:35.474512 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:35.474475    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9"]
Oct 30 22:11:35.475019 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:11:35.474988    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a031d03_d200_4477_9a61_8385a063abf1.slice/crio-94ea4c79c4bbd46f9c1e586b40492d4bfa2e24675eb3be87cdfaba4544709929 WatchSource:0}: Error finding container 94ea4c79c4bbd46f9c1e586b40492d4bfa2e24675eb3be87cdfaba4544709929: Status 404 returned error can't find the container with id 94ea4c79c4bbd46f9c1e586b40492d4bfa2e24675eb3be87cdfaba4544709929
Oct 30 22:11:35.711232 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:35.711201    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c90f6ae-36df-4896-afa5-8859e01044c0" path="/var/lib/kubelet/pods/1c90f6ae-36df-4896-afa5-8859e01044c0/volumes"
Oct 30 22:11:35.711664 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:35.711643    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cae946a0-125e-4832-96eb-8b824f689a88" path="/var/lib/kubelet/pods/cae946a0-125e-4832-96eb-8b824f689a88/volumes"
Oct 30 22:11:35.906654 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:35.906564    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9" event={"ID":"8a031d03-d200-4477-9a61-8385a063abf1","Type":"ContainerStarted","Data":"94ea4c79c4bbd46f9c1e586b40492d4bfa2e24675eb3be87cdfaba4544709929"}
Oct 30 22:11:36.762289 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:36.762243    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc" podUID="f6a9f73b-bb1f-4270-a38d-3703c46fae07" containerName="main" containerID="cri-o://ce3a1be70a92bba361c420e743a45b06804c17f31f1eb76ea630c8fd68f74b9c" gracePeriod=45
Oct 30 22:11:36.915996 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:36.915956    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9" event={"ID":"8a031d03-d200-4477-9a61-8385a063abf1","Type":"ContainerStarted","Data":"4c8e628dd24c3c80a37135c058fbb3db142aa8808efa98b4dd89e9a2ca981f5b"}
Oct 30 22:11:36.916180 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:36.916049    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9"
Oct 30 22:11:36.917779 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:36.917753    5087 generic.go:356] "Generic (PLEG): container finished" podID="f6a9f73b-bb1f-4270-a38d-3703c46fae07" containerID="ce3a1be70a92bba361c420e743a45b06804c17f31f1eb76ea630c8fd68f74b9c" exitCode=0
Oct 30 22:11:36.917917 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:36.917787    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc" event={"ID":"f6a9f73b-bb1f-4270-a38d-3703c46fae07","Type":"ContainerDied","Data":"ce3a1be70a92bba361c420e743a45b06804c17f31f1eb76ea630c8fd68f74b9c"}
Oct 30 22:11:37.034879 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:37.034832    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9" podStartSLOduration=2.215673401 podStartE2EDuration="3.034816249s" totalImagesPullingTime="819.142848ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:37.033024813 +0000 UTC m=+2291.507528791" watchObservedRunningTime="2026-10-30 22:11:37.034816249 +0000 UTC m=+2291.509320228"
Oct 30 22:11:37.064083 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:37.064050    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 22:11:37.136327 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:37.136298    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6a9f73b-bb1f-4270-a38d-3703c46fae07-tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a9f73b-bb1f-4270-a38d-3703c46fae07-tls-certs\") pod \"f6a9f73b-bb1f-4270-a38d-3703c46fae07\" (UID: \"f6a9f73b-bb1f-4270-a38d-3703c46fae07\") "
Oct 30 22:11:37.136515 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:37.136403    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6a9f73b-bb1f-4270-a38d-3703c46fae07-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a9f73b-bb1f-4270-a38d-3703c46fae07-tmp-dir\") pod \"f6a9f73b-bb1f-4270-a38d-3703c46fae07\" (UID: \"f6a9f73b-bb1f-4270-a38d-3703c46fae07\") "
Oct 30 22:11:37.136515 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:37.136432    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6a9f73b-bb1f-4270-a38d-3703c46fae07-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a9f73b-bb1f-4270-a38d-3703c46fae07-dshm\") pod \"f6a9f73b-bb1f-4270-a38d-3703c46fae07\" (UID: \"f6a9f73b-bb1f-4270-a38d-3703c46fae07\") "
Oct 30 22:11:37.136515 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:37.136456    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6a9f73b-bb1f-4270-a38d-3703c46fae07-home\" (UniqueName: \"kubernetes.io/empty-dir/f6a9f73b-bb1f-4270-a38d-3703c46fae07-home\") pod \"f6a9f73b-bb1f-4270-a38d-3703c46fae07\" (UID: \"f6a9f73b-bb1f-4270-a38d-3703c46fae07\") "
Oct 30 22:11:37.136694 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:37.136516    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6a9f73b-bb1f-4270-a38d-3703c46fae07-kube-api-access-gfksp\" (UniqueName: \"kubernetes.io/projected/f6a9f73b-bb1f-4270-a38d-3703c46fae07-kube-api-access-gfksp\") pod \"f6a9f73b-bb1f-4270-a38d-3703c46fae07\" (UID: \"f6a9f73b-bb1f-4270-a38d-3703c46fae07\") "
Oct 30 22:11:37.136694 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:37.136571    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6a9f73b-bb1f-4270-a38d-3703c46fae07-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a9f73b-bb1f-4270-a38d-3703c46fae07-model-cache\") pod \"f6a9f73b-bb1f-4270-a38d-3703c46fae07\" (UID: \"f6a9f73b-bb1f-4270-a38d-3703c46fae07\") "
Oct 30 22:11:37.136795 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:37.136699    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a9f73b-bb1f-4270-a38d-3703c46fae07-tmp-dir" pod "f6a9f73b-bb1f-4270-a38d-3703c46fae07" (UID: "f6a9f73b-bb1f-4270-a38d-3703c46fae07"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:37.137168 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:37.136935    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a9f73b-bb1f-4270-a38d-3703c46fae07-model-cache" pod "f6a9f73b-bb1f-4270-a38d-3703c46fae07" (UID: "f6a9f73b-bb1f-4270-a38d-3703c46fae07"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:37.137168 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:37.136955    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a9f73b-bb1f-4270-a38d-3703c46fae07-home" pod "f6a9f73b-bb1f-4270-a38d-3703c46fae07" (UID: "f6a9f73b-bb1f-4270-a38d-3703c46fae07"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:37.137168 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:37.137068    5087 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6a9f73b-bb1f-4270-a38d-3703c46fae07-model-cache\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:37.137168 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:37.137088    5087 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6a9f73b-bb1f-4270-a38d-3703c46fae07-tmp-dir\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:37.137168 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:37.137102    5087 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6a9f73b-bb1f-4270-a38d-3703c46fae07-home\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:37.138970 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:37.138948    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a9f73b-bb1f-4270-a38d-3703c46fae07-tls-certs" pod "f6a9f73b-bb1f-4270-a38d-3703c46fae07" (UID: "f6a9f73b-bb1f-4270-a38d-3703c46fae07"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:37.139326 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:37.139307    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6a9f73b-bb1f-4270-a38d-3703c46fae07-dshm" pod "f6a9f73b-bb1f-4270-a38d-3703c46fae07" (UID: "f6a9f73b-bb1f-4270-a38d-3703c46fae07"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:11:37.139587 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:37.139568    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a9f73b-bb1f-4270-a38d-3703c46fae07-kube-api-access-gfksp" pod "f6a9f73b-bb1f-4270-a38d-3703c46fae07" (UID: "f6a9f73b-bb1f-4270-a38d-3703c46fae07"). InnerVolumeSpecName "kube-api-access-gfksp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:37.238573 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:37.238536    5087 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6a9f73b-bb1f-4270-a38d-3703c46fae07-dshm\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:37.238573 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:37.238575    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfksp\" (UniqueName: \"kubernetes.io/projected/f6a9f73b-bb1f-4270-a38d-3703c46fae07-kube-api-access-gfksp\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:37.238783 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:37.238592    5087 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6a9f73b-bb1f-4270-a38d-3703c46fae07-tls-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:37.711670 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:37.711642    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh"]
Oct 30 22:11:37.926595 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:37.926565    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"
Oct 30 22:11:37.926595 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:37.926589    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc" event={"ID":"f6a9f73b-bb1f-4270-a38d-3703c46fae07","Type":"ContainerDied","Data":"1d26303041bb547d1b3a13a4d85d6478d42f6d7a06311c902357bda5b6660fd4"}
Oct 30 22:11:37.927106 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:37.926637    5087 scope.go:122] "RemoveContainer" containerID="ce3a1be70a92bba361c420e743a45b06804c17f31f1eb76ea630c8fd68f74b9c"
Oct 30 22:11:38.090930 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.090900    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"]
Oct 30 22:11:38.096630 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.096603    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-9b51e1/simulator-w5-9b51e1-kserve-76cbc88bb9-g6pjc"]
Oct 30 22:11:38.342905 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.342822    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"]
Oct 30 22:11:38.343417 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.343399    5087 cpu_manager.go:414] "containerMap: removing container" podUID="f6a9f73b-bb1f-4270-a38d-3703c46fae07" containerName="main"
Oct 30 22:11:38.343417 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.343417    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6a9f73b-bb1f-4270-a38d-3703c46fae07" containerName="main"
Oct 30 22:11:38.343583 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.343533    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6a9f73b-bb1f-4270-a38d-3703c46fae07" containerName="main"
Oct 30 22:11:38.348257 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.348217    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:38.352814 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.352786    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"]
Oct 30 22:11:38.357275 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.357250    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:38.366537 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.366506    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"]
Oct 30 22:11:38.370682 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.370661    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:38.400614 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.400587    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-xn2np\""
Oct 30 22:11:38.404365 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.404319    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"]
Oct 30 22:11:38.448417 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.443392    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"]
Oct 30 22:11:38.455136 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.455095    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076fae6b-35bc-408e-aec8-e45316a2e9fa-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-44qk9\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:38.455971 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.455946    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6d8084-9482-438d-b5e5-3f36442423d2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pwfjh\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:38.456140 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.456120    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pptz4\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:38.456286 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.456269    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pwfjh\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:38.456476 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.456456    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076fae6b-35bc-408e-aec8-e45316a2e9fa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-44qk9\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:38.456616 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.456595    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-44qk9\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:38.456741 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.456725    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxdzv\" (UniqueName: \"kubernetes.io/projected/55c06e30-cb56-47cb-ae02-fc44e936bc48-kube-api-access-dxdzv\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pptz4\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:38.456918 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.456889    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw5dn\" (UniqueName: \"kubernetes.io/projected/076fae6b-35bc-408e-aec8-e45316a2e9fa-kube-api-access-gw5dn\") pod \"maas-default-gateway-openshift-default-74854d8ccf-44qk9\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:38.457061 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.457044    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pwfjh\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:38.458094 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.458069    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c06e30-cb56-47cb-ae02-fc44e936bc48-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pptz4\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:38.458188 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.458151    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-44qk9\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:38.458265 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.458185    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pptz4\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:38.458415 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.458260    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-44qk9\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:38.458579 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.458560    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pwfjh\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:38.458644 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.458615    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6d8084-9482-438d-b5e5-3f36442423d2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pwfjh\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:38.458712 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.458661    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pptz4\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:38.458712 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.458692    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pwfjh\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:38.458829 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.458729    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55c06e30-cb56-47cb-ae02-fc44e936bc48-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pptz4\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:38.458887 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.458864    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pptz4\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:38.459177 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.458931    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pwfjh\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:38.459177 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.458970    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pptz4\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:38.459177 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.459032    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb5kb\" (UniqueName: \"kubernetes.io/projected/9c6d8084-9482-438d-b5e5-3f36442423d2-kube-api-access-hb5kb\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pwfjh\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:38.459177 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.459115    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6d8084-9482-438d-b5e5-3f36442423d2-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pwfjh\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:38.459177 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.459150    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-44qk9\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:38.459504 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.459186    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-44qk9\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:38.459504 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.459221    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c06e30-cb56-47cb-ae02-fc44e936bc48-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pptz4\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:38.459504 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.459385    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076fae6b-35bc-408e-aec8-e45316a2e9fa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-44qk9\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:38.469945 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.469914    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"]
Oct 30 22:11:38.560538 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.560442    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pptz4\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:38.560745 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.560563    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-44qk9\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:38.560745 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.560605    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pwfjh\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:38.560745 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.560634    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6d8084-9482-438d-b5e5-3f36442423d2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pwfjh\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:38.560745 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.560668    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pptz4\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:38.560745 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.560694    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pwfjh\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:38.560745 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.560722    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55c06e30-cb56-47cb-ae02-fc44e936bc48-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pptz4\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:38.561078 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.560761    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pptz4\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:38.561078 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.560792    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pwfjh\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:38.561078 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.560821    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pptz4\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:38.561078 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.560886    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb5kb\" (UniqueName: \"kubernetes.io/projected/9c6d8084-9482-438d-b5e5-3f36442423d2-kube-api-access-hb5kb\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pwfjh\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:38.561078 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.560924    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pptz4\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:38.561078 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.561025    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pwfjh\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:38.561078 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.561032    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-44qk9\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:38.561078 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.560934    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6d8084-9482-438d-b5e5-3f36442423d2-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pwfjh\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:38.561533 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.561069    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pptz4\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:38.561533 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.561101    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-44qk9\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:38.561533 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.561144    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-44qk9\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:38.561533 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.561174    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pwfjh\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:38.561533 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.561186    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c06e30-cb56-47cb-ae02-fc44e936bc48-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pptz4\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:38.561533 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.561250    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076fae6b-35bc-408e-aec8-e45316a2e9fa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-44qk9\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:38.561533 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.561266    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pwfjh\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:38.561533 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.561328    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076fae6b-35bc-408e-aec8-e45316a2e9fa-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-44qk9\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:38.561533 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.561410    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6d8084-9482-438d-b5e5-3f36442423d2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pwfjh\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:38.561533 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.561420    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pptz4\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:38.561533 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.561437    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pptz4\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:38.561533 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.561476    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pwfjh\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:38.561533 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.561513    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076fae6b-35bc-408e-aec8-e45316a2e9fa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-44qk9\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:38.561533 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.561543    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-44qk9\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:38.562281 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.561577    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxdzv\" (UniqueName: \"kubernetes.io/projected/55c06e30-cb56-47cb-ae02-fc44e936bc48-kube-api-access-dxdzv\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pptz4\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:38.562281 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.561606    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw5dn\" (UniqueName: \"kubernetes.io/projected/076fae6b-35bc-408e-aec8-e45316a2e9fa-kube-api-access-gw5dn\") pod \"maas-default-gateway-openshift-default-74854d8ccf-44qk9\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:38.562281 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.561650    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pwfjh\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:38.562281 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.561689    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c06e30-cb56-47cb-ae02-fc44e936bc48-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pptz4\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:38.562281 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.561732    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-44qk9\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:38.562607 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.562370    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pptz4\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:38.562829 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.562795    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6d8084-9482-438d-b5e5-3f36442423d2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pwfjh\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:38.562924 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.562865    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pwfjh\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:38.563288 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.563264    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-44qk9\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:38.563397 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.563370    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-44qk9\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:38.563516 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.563490    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-44qk9\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:38.563768 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.563738    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c06e30-cb56-47cb-ae02-fc44e936bc48-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pptz4\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:38.564034 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.564011    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076fae6b-35bc-408e-aec8-e45316a2e9fa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-44qk9\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:38.564664 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.564636    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pptz4\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:38.565596 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.565568    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pwfjh\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:38.566096 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.566072    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-44qk9\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:38.566446 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.566425    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076fae6b-35bc-408e-aec8-e45316a2e9fa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-44qk9\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:38.566536 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.566432    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c06e30-cb56-47cb-ae02-fc44e936bc48-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pptz4\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:38.566696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.566679    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6d8084-9482-438d-b5e5-3f36442423d2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pwfjh\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:38.618154 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.618084    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6d8084-9482-438d-b5e5-3f36442423d2-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pwfjh\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:38.618273 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.618180    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55c06e30-cb56-47cb-ae02-fc44e936bc48-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pptz4\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:38.619167 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.619139    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb5kb\" (UniqueName: \"kubernetes.io/projected/9c6d8084-9482-438d-b5e5-3f36442423d2-kube-api-access-hb5kb\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pwfjh\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:38.619287 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.619231    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076fae6b-35bc-408e-aec8-e45316a2e9fa-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-44qk9\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:38.619287 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.619255    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxdzv\" (UniqueName: \"kubernetes.io/projected/55c06e30-cb56-47cb-ae02-fc44e936bc48-kube-api-access-dxdzv\") pod \"maas-default-gateway-openshift-default-74854d8ccf-pptz4\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:38.619406 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.619388    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw5dn\" (UniqueName: \"kubernetes.io/projected/076fae6b-35bc-408e-aec8-e45316a2e9fa-kube-api-access-gw5dn\") pod \"maas-default-gateway-openshift-default-74854d8ccf-44qk9\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:38.670522 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.670490    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:38.684254 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.684228    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:38.695368 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:38.691476    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:39.112279 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:39.111998    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"]
Oct 30 22:11:39.113496 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:11:39.113470    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod076fae6b_35bc_408e_aec8_e45316a2e9fa.slice/crio-b585cf964dff1b63c647fc25c1ccda39fba56c2fde4e31a301793165ed657537 WatchSource:0}: Error finding container b585cf964dff1b63c647fc25c1ccda39fba56c2fde4e31a301793165ed657537: Status 404 returned error can't find the container with id b585cf964dff1b63c647fc25c1ccda39fba56c2fde4e31a301793165ed657537
Oct 30 22:11:39.115284 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:39.115267    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"]
Oct 30 22:11:39.116144 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:39.116114    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:11:39.116209 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:39.116197    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:11:39.116245 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:39.116229    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:11:39.118674 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:11:39.118652    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c6d8084_9482_438d_b5e5_3f36442423d2.slice/crio-483415df9aaae7530af93e69decc3564fa76351faf07d425c841e66bba087ff1 WatchSource:0}: Error finding container 483415df9aaae7530af93e69decc3564fa76351faf07d425c841e66bba087ff1: Status 404 returned error can't find the container with id 483415df9aaae7530af93e69decc3564fa76351faf07d425c841e66bba087ff1
Oct 30 22:11:39.121316 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:39.121285    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:11:39.121444 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:39.121421    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:11:39.121514 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:39.121476    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:11:39.176608 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:39.176560    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"]
Oct 30 22:11:39.216757 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:39.216710    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:11:39.216916 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:39.216810    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:11:39.216916 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:39.216856    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:11:39.713439 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:39.713397    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6a9f73b-bb1f-4270-a38d-3703c46fae07" path="/var/lib/kubelet/pods/f6a9f73b-bb1f-4270-a38d-3703c46fae07/volumes"
Oct 30 22:11:39.845561 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:39.845498    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6df3ba0e-84d7646477-6qgzr"
Oct 30 22:11:39.847019 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:39.846995    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6df3ba0e-7455bf6cc9-mcc6n"
Oct 30 22:11:40.320006 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:40.315835    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh" event={"ID":"9c6d8084-9482-438d-b5e5-3f36442423d2","Type":"ContainerStarted","Data":"3ac6e46a230b2d2899daa347f3b1f02aaf3b31b6d9150fb29535dfb0b769fe5b"}
Oct 30 22:11:40.320006 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:40.315910    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh" event={"ID":"9c6d8084-9482-438d-b5e5-3f36442423d2","Type":"ContainerStarted","Data":"483415df9aaae7530af93e69decc3564fa76351faf07d425c841e66bba087ff1"}
Oct 30 22:11:40.331177 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:40.331135    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9" event={"ID":"076fae6b-35bc-408e-aec8-e45316a2e9fa","Type":"ContainerStarted","Data":"852fe6e3f484f446ece2ce23111cb2653b54947cf8d9a48e09a0329990c163b9"}
Oct 30 22:11:40.331332 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:40.331202    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9" event={"ID":"076fae6b-35bc-408e-aec8-e45316a2e9fa","Type":"ContainerStarted","Data":"b585cf964dff1b63c647fc25c1ccda39fba56c2fde4e31a301793165ed657537"}
Oct 30 22:11:40.333514 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:40.333483    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4" event={"ID":"55c06e30-cb56-47cb-ae02-fc44e936bc48","Type":"ContainerStarted","Data":"75d5db3719f7c73fb7eab8feaa5664fa801998f4a88b7fb716630c3684d04245"}
Oct 30 22:11:40.333644 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:40.333535    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4" event={"ID":"55c06e30-cb56-47cb-ae02-fc44e936bc48","Type":"ContainerStarted","Data":"95e9d0a989c284e56183106807939cc3c626a15a7f8233f7f3faf10b6856cf9e"}
Oct 30 22:11:40.483378 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:40.482980    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh" podStartSLOduration=2.482955155 podStartE2EDuration="2.482955155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:40.470874063 +0000 UTC m=+2294.945378033" watchObservedRunningTime="2026-10-30 22:11:40.482955155 +0000 UTC m=+2294.957459135"
Oct 30 22:11:40.615464 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:40.600139    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9" podStartSLOduration=2.600123015 podStartE2EDuration="2.600123015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:40.598610431 +0000 UTC m=+2295.073114411" watchObservedRunningTime="2026-10-30 22:11:40.600123015 +0000 UTC m=+2295.074626994"
Oct 30 22:11:40.671757 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:40.671721    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:40.679514 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:40.679481    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-44qk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 22:11:40.679754 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:40.679728    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9" podUID="076fae6b-35bc-408e-aec8-e45316a2e9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 22:11:40.691140 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:40.691110    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:40.691538 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:40.691515    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-pwfjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:11:40.691701 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:40.691681    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh" podUID="9c6d8084-9482-438d-b5e5-3f36442423d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 22:11:40.693207 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:40.693076    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:40.693333 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:40.693146    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-pptz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 22:11:40.693648 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:40.693629    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4" podUID="55c06e30-cb56-47cb-ae02-fc44e936bc48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 22:11:40.762388 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:40.732173    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4" podStartSLOduration=2.732152329 podStartE2EDuration="2.732152329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:40.730730249 +0000 UTC m=+2295.205234232" watchObservedRunningTime="2026-10-30 22:11:40.732152329 +0000 UTC m=+2295.206656310"
Oct 30 22:11:41.675379 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:41.674779    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-44qk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 22:11:41.675379 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:41.674845    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9" podUID="076fae6b-35bc-408e-aec8-e45316a2e9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 22:11:41.690375 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:41.687751    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-pwfjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:11:41.690375 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:41.687810    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh" podUID="9c6d8084-9482-438d-b5e5-3f36442423d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 22:11:41.712375 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:41.693737    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-pptz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 22:11:41.712375 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:41.693815    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4" podUID="55c06e30-cb56-47cb-ae02-fc44e936bc48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 22:11:42.676821 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:42.676776    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-44qk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 22:11:42.677477 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:42.677443    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9" podUID="076fae6b-35bc-408e-aec8-e45316a2e9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 22:11:42.689780 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:42.689746    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-pwfjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:11:42.690010 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:42.689985    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh" podUID="9c6d8084-9482-438d-b5e5-3f36442423d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 22:11:42.693735 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:42.693709    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-pptz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 22:11:42.693904 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:42.693881    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4" podUID="55c06e30-cb56-47cb-ae02-fc44e936bc48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 22:11:42.713853 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:42.713815    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh" podUID="ad32b3aa-9229-422b-b7d5-6d2bd8531138" containerName="payload-processing" containerID="cri-o://6eb8804136865c7eba0d9d31fedceca7c9ce278972d137ea5c02656a9a093f36" gracePeriod=25
Oct 30 22:11:42.933948 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:42.933863    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9"
Oct 30 22:11:43.409377 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:43.407557    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v" podUID="88a9407e-afe4-4e0b-af28-20226fb487c0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:11:43.567492 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:43.567450    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw" podUID="14ebb851-d1f0-43a8-8351-9fbe3e5a4e71" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:11:43.671685 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:43.671592    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-44qk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 22:11:43.681090 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:43.681048    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9" podUID="076fae6b-35bc-408e-aec8-e45316a2e9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 22:11:43.687861 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:43.687827    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-pwfjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:11:43.688084 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:43.688058    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh" podUID="9c6d8084-9482-438d-b5e5-3f36442423d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 22:11:43.701387 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:43.700772    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-pptz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 22:11:43.701387 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:43.700823    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4" podUID="55c06e30-cb56-47cb-ae02-fc44e936bc48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 22:11:44.029148 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.029121    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f87c2fdb-jrs24"]
Oct 30 22:11:44.034167 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.033738    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c2fdb-jrs24"
Oct 30 22:11:44.120380 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.120225    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:11:44.127405 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.121042    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:11:44.161391 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.135495    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82c035ad-wl8nc"]
Oct 30 22:11:44.192708 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.182678    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c035ad-wl8nc"
Oct 30 22:11:44.192708 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.186147    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40934d2d-712a-4070-935e-cbeb5120c309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f87c2fdb-jrs24\" (UID: \"40934d2d-712a-4070-935e-cbeb5120c309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c2fdb-jrs24"
Oct 30 22:11:44.192708 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.186281    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wx6b\" (UniqueName: \"kubernetes.io/projected/40934d2d-712a-4070-935e-cbeb5120c309-kube-api-access-2wx6b\") pod \"maas-api-revoke-sub-f87c2fdb-jrs24\" (UID: \"40934d2d-712a-4070-935e-cbeb5120c309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c2fdb-jrs24"
Oct 30 22:11:44.192708 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.186334    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40934d2d-712a-4070-935e-cbeb5120c309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f87c2fdb-jrs24\" (UID: \"40934d2d-712a-4070-935e-cbeb5120c309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c2fdb-jrs24"
Oct 30 22:11:44.325302 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.288743    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44lvl\" (UniqueName: \"kubernetes.io/projected/36ffceb8-7a93-46f7-897d-4a3a72b2864f-kube-api-access-44lvl\") pod \"maas-api-revoke-sub-82c035ad-wl8nc\" (UID: \"36ffceb8-7a93-46f7-897d-4a3a72b2864f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c035ad-wl8nc"
Oct 30 22:11:44.325302 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.288786    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ffceb8-7a93-46f7-897d-4a3a72b2864f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82c035ad-wl8nc\" (UID: \"36ffceb8-7a93-46f7-897d-4a3a72b2864f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c035ad-wl8nc"
Oct 30 22:11:44.325302 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.288842    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40934d2d-712a-4070-935e-cbeb5120c309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f87c2fdb-jrs24\" (UID: \"40934d2d-712a-4070-935e-cbeb5120c309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c2fdb-jrs24"
Oct 30 22:11:44.325302 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.288956    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ffceb8-7a93-46f7-897d-4a3a72b2864f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82c035ad-wl8nc\" (UID: \"36ffceb8-7a93-46f7-897d-4a3a72b2864f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c035ad-wl8nc"
Oct 30 22:11:44.325302 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.288992    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wx6b\" (UniqueName: \"kubernetes.io/projected/40934d2d-712a-4070-935e-cbeb5120c309-kube-api-access-2wx6b\") pod \"maas-api-revoke-sub-f87c2fdb-jrs24\" (UID: \"40934d2d-712a-4070-935e-cbeb5120c309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c2fdb-jrs24"
Oct 30 22:11:44.325302 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.289035    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40934d2d-712a-4070-935e-cbeb5120c309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f87c2fdb-jrs24\" (UID: \"40934d2d-712a-4070-935e-cbeb5120c309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c2fdb-jrs24"
Oct 30 22:11:44.337393 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.330054    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40934d2d-712a-4070-935e-cbeb5120c309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f87c2fdb-jrs24\" (UID: \"40934d2d-712a-4070-935e-cbeb5120c309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c2fdb-jrs24"
Oct 30 22:11:44.393373 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.393301    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44lvl\" (UniqueName: \"kubernetes.io/projected/36ffceb8-7a93-46f7-897d-4a3a72b2864f-kube-api-access-44lvl\") pod \"maas-api-revoke-sub-82c035ad-wl8nc\" (UID: \"36ffceb8-7a93-46f7-897d-4a3a72b2864f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c035ad-wl8nc"
Oct 30 22:11:44.434695 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.434190    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ffceb8-7a93-46f7-897d-4a3a72b2864f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82c035ad-wl8nc\" (UID: \"36ffceb8-7a93-46f7-897d-4a3a72b2864f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c035ad-wl8nc"
Oct 30 22:11:44.475747 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.442167    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ffceb8-7a93-46f7-897d-4a3a72b2864f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82c035ad-wl8nc\" (UID: \"36ffceb8-7a93-46f7-897d-4a3a72b2864f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c035ad-wl8nc"
Oct 30 22:11:44.573850 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.497869    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ffceb8-7a93-46f7-897d-4a3a72b2864f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82c035ad-wl8nc\" (UID: \"36ffceb8-7a93-46f7-897d-4a3a72b2864f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c035ad-wl8nc"
Oct 30 22:11:44.573850 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.509400    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f87c2fdb-jrs24"]
Oct 30 22:11:44.573850 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.572338    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40934d2d-712a-4070-935e-cbeb5120c309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f87c2fdb-jrs24\" (UID: \"40934d2d-712a-4070-935e-cbeb5120c309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c2fdb-jrs24"
Oct 30 22:11:44.573850 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.572964    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wx6b\" (UniqueName: \"kubernetes.io/projected/40934d2d-712a-4070-935e-cbeb5120c309-kube-api-access-2wx6b\") pod \"maas-api-revoke-sub-f87c2fdb-jrs24\" (UID: \"40934d2d-712a-4070-935e-cbeb5120c309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c2fdb-jrs24"
Oct 30 22:11:44.606475 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.606235    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c2fdb-jrs24"
Oct 30 22:11:44.610034 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.609455    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82c035ad-wl8nc"]
Oct 30 22:11:44.734477 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.692901    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-pptz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 22:11:44.734477 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.692959    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4" podUID="55c06e30-cb56-47cb-ae02-fc44e936bc48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 22:11:44.734477 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.693386    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-pwfjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:11:44.734477 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.693425    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh" podUID="9c6d8084-9482-438d-b5e5-3f36442423d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 22:11:44.734477 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.708647    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-44qk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 22:11:44.734477 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.708698    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9" podUID="076fae6b-35bc-408e-aec8-e45316a2e9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 22:11:44.734477 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.710909    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ffceb8-7a93-46f7-897d-4a3a72b2864f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82c035ad-wl8nc\" (UID: \"36ffceb8-7a93-46f7-897d-4a3a72b2864f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c035ad-wl8nc"
Oct 30 22:11:44.859724 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.841332    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44lvl\" (UniqueName: \"kubernetes.io/projected/36ffceb8-7a93-46f7-897d-4a3a72b2864f-kube-api-access-44lvl\") pod \"maas-api-revoke-sub-82c035ad-wl8nc\" (UID: \"36ffceb8-7a93-46f7-897d-4a3a72b2864f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c035ad-wl8nc"
Oct 30 22:11:44.915381 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.892733    5087 generic.go:356] "Generic (PLEG): container finished" podID="14ebb851-d1f0-43a8-8351-9fbe3e5a4e71" containerID="e01ff47a8e18f1b2569c9ab88e80f25759536c33cea399cf3d8e7178fc6cdcf9" exitCode=0
Oct 30 22:11:44.915381 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.892870    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw" event={"ID":"14ebb851-d1f0-43a8-8351-9fbe3e5a4e71","Type":"ContainerDied","Data":"e01ff47a8e18f1b2569c9ab88e80f25759536c33cea399cf3d8e7178fc6cdcf9"}
Oct 30 22:11:44.926237 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.926195    5087 generic.go:356] "Generic (PLEG): container finished" podID="88a9407e-afe4-4e0b-af28-20226fb487c0" containerID="bf880dd59f6d51e81b9cf036e97aae707e3315cb42adf4b9002491dccd609583" exitCode=0
Oct 30 22:11:44.933870 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:44.926663    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v" event={"ID":"88a9407e-afe4-4e0b-af28-20226fb487c0","Type":"ContainerDied","Data":"bf880dd59f6d51e81b9cf036e97aae707e3315cb42adf4b9002491dccd609583"}
Oct 30 22:11:45.057981 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.057944    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c035ad-wl8nc"
Oct 30 22:11:45.162664 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.161039    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:45.220967 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.220629    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw"
Oct 30 22:11:45.422985 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.421785    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14ebb851-d1f0-43a8-8351-9fbe3e5a4e71-kube-api-access-7djzj\" (UniqueName: \"kubernetes.io/projected/14ebb851-d1f0-43a8-8351-9fbe3e5a4e71-kube-api-access-7djzj\") pod \"14ebb851-d1f0-43a8-8351-9fbe3e5a4e71\" (UID: \"14ebb851-d1f0-43a8-8351-9fbe3e5a4e71\") "
Oct 30 22:11:45.422985 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.421930    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14ebb851-d1f0-43a8-8351-9fbe3e5a4e71-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14ebb851-d1f0-43a8-8351-9fbe3e5a4e71-plugins-config-volume\") pod \"14ebb851-d1f0-43a8-8351-9fbe3e5a4e71\" (UID: \"14ebb851-d1f0-43a8-8351-9fbe3e5a4e71\") "
Oct 30 22:11:45.423772 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.423406    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14ebb851-d1f0-43a8-8351-9fbe3e5a4e71-plugins-config-volume" pod "14ebb851-d1f0-43a8-8351-9fbe3e5a4e71" (UID: "14ebb851-d1f0-43a8-8351-9fbe3e5a4e71"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:45.459679 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.439545    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:45.510376 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.464455    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:45.510376 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.506933    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14ebb851-d1f0-43a8-8351-9fbe3e5a4e71-kube-api-access-7djzj" pod "14ebb851-d1f0-43a8-8351-9fbe3e5a4e71" (UID: "14ebb851-d1f0-43a8-8351-9fbe3e5a4e71"). InnerVolumeSpecName "kube-api-access-7djzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:45.532813 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.532776    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7djzj\" (UniqueName: \"kubernetes.io/projected/14ebb851-d1f0-43a8-8351-9fbe3e5a4e71-kube-api-access-7djzj\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:45.533054 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.533034    5087 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14ebb851-d1f0-43a8-8351-9fbe3e5a4e71-plugins-config-volume\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:45.533210 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.533181    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:45.534140 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.534117    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded"
Oct 30 22:11:45.604795 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.595945    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:45.696204 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.695406    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-44qk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 22:11:45.696204 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.695462    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9" podUID="076fae6b-35bc-408e-aec8-e45316a2e9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 22:11:45.698250 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.698002    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-pwfjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:11:45.698250 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.698049    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh" podUID="9c6d8084-9482-438d-b5e5-3f36442423d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 22:11:45.698444 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.698396    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-pptz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 22:11:45.698444 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.698436    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4" podUID="55c06e30-cb56-47cb-ae02-fc44e936bc48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 22:11:45.809439 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.806894    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v"
Oct 30 22:11:45.841238 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.841210    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88a9407e-afe4-4e0b-af28-20226fb487c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a9407e-afe4-4e0b-af28-20226fb487c0-plugins-config-volume\") pod \"88a9407e-afe4-4e0b-af28-20226fb487c0\" (UID: \"88a9407e-afe4-4e0b-af28-20226fb487c0\") "
Oct 30 22:11:45.841447 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.841280    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88a9407e-afe4-4e0b-af28-20226fb487c0-kube-api-access-l827s\" (UniqueName: \"kubernetes.io/projected/88a9407e-afe4-4e0b-af28-20226fb487c0-kube-api-access-l827s\") pod \"88a9407e-afe4-4e0b-af28-20226fb487c0\" (UID: \"88a9407e-afe4-4e0b-af28-20226fb487c0\") "
Oct 30 22:11:45.841715 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.841683    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88a9407e-afe4-4e0b-af28-20226fb487c0-plugins-config-volume" pod "88a9407e-afe4-4e0b-af28-20226fb487c0" (UID: "88a9407e-afe4-4e0b-af28-20226fb487c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:45.841842 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.841823    5087 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88a9407e-afe4-4e0b-af28-20226fb487c0-plugins-config-volume\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:45.843573 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.843550    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a9407e-afe4-4e0b-af28-20226fb487c0-kube-api-access-l827s" pod "88a9407e-afe4-4e0b-af28-20226fb487c0" (UID: "88a9407e-afe4-4e0b-af28-20226fb487c0"). InnerVolumeSpecName "kube-api-access-l827s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:45.870674 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.870651    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82c035ad-wl8nc"]
Oct 30 22:11:45.873336 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:11:45.873306    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ffceb8_7a93_46f7_897d_4a3a72b2864f.slice/crio-571c04f4351c2528bdc687959dd2726872ce6b91b96680e3913461284d6f5654 WatchSource:0}: Error finding container 571c04f4351c2528bdc687959dd2726872ce6b91b96680e3913461284d6f5654: Status 404 returned error can't find the container with id 571c04f4351c2528bdc687959dd2726872ce6b91b96680e3913461284d6f5654
Oct 30 22:11:45.943374 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.943319    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l827s\" (UniqueName: \"kubernetes.io/projected/88a9407e-afe4-4e0b-af28-20226fb487c0-kube-api-access-l827s\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:45.951423 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.951234    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw" event={"ID":"14ebb851-d1f0-43a8-8351-9fbe3e5a4e71","Type":"ContainerDied","Data":"77b52e76f9a3cec4f2e6ef708417b86da7434de106585a15f039788fb1f6351b"}
Oct 30 22:11:45.951423 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.951378    5087 scope.go:122] "RemoveContainer" containerID="e01ff47a8e18f1b2569c9ab88e80f25759536c33cea399cf3d8e7178fc6cdcf9"
Oct 30 22:11:45.952501 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.951565    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw"
Oct 30 22:11:45.955587 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.955523    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c035ad-wl8nc" event={"ID":"36ffceb8-7a93-46f7-897d-4a3a72b2864f","Type":"ContainerStarted","Data":"571c04f4351c2528bdc687959dd2726872ce6b91b96680e3913461284d6f5654"}
Oct 30 22:11:45.957768 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.957746    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v" event={"ID":"88a9407e-afe4-4e0b-af28-20226fb487c0","Type":"ContainerDied","Data":"663ca1c8ad544c23d90bdc4a7fab70687736420ae5cfcf69ef394fe0c6707351"}
Oct 30 22:11:45.958079 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.957919    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v"
Oct 30 22:11:45.976569 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:45.975887    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f87c2fdb-jrs24"]
Oct 30 22:11:46.014584 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:46.014449    5087 scope.go:122] "RemoveContainer" containerID="bf880dd59f6d51e81b9cf036e97aae707e3315cb42adf4b9002491dccd609583"
Oct 30 22:11:46.170484 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:46.170446    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw"]
Oct 30 22:11:46.175080 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:46.175052    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-dda9f5-75fcc89f49-bh6xw"]
Oct 30 22:11:46.283556 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:46.282623    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v"]
Oct 30 22:11:46.286821 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:46.286789    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-dda9f5-5cb698d9c9-hj25v"]
Oct 30 22:11:46.672211 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:46.672175    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-44qk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body=
Oct 30 22:11:46.672541 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:46.672508    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9" podUID="076fae6b-35bc-408e-aec8-e45316a2e9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused"
Oct 30 22:11:46.686581 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:46.686540    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-pwfjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 30 22:11:46.686727 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:46.686602    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh" podUID="9c6d8084-9482-438d-b5e5-3f36442423d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 30 22:11:46.692864 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:46.692830    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-pptz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 30 22:11:46.693001 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:46.692891    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4" podUID="55c06e30-cb56-47cb-ae02-fc44e936bc48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 30 22:11:46.973808 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:46.973671    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c035ad-wl8nc" event={"ID":"36ffceb8-7a93-46f7-897d-4a3a72b2864f","Type":"ContainerStarted","Data":"afabac5215053f697d1ef8032d2ce9c99e1a08643d2b591c0d78ca58adf1bac4"}
Oct 30 22:11:46.977520 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:46.977491    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c2fdb-jrs24" event={"ID":"40934d2d-712a-4070-935e-cbeb5120c309","Type":"ContainerStarted","Data":"8e17283b9c1288f08f02f1408f8f968f6503aa741e018c1e9820083646d8e5d2"}
Oct 30 22:11:46.977735 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:46.977716    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c2fdb-jrs24" event={"ID":"40934d2d-712a-4070-935e-cbeb5120c309","Type":"ContainerStarted","Data":"702127f52d027ed8455d9a34221ecad851989911dc8cd53638d23f5d8621dd54"}
Oct 30 22:11:47.093431 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:47.093337    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c035ad-wl8nc" podStartSLOduration=3.093318367 podStartE2EDuration="3.093318367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:47.089996561 +0000 UTC m=+2301.564500541" watchObservedRunningTime="2026-10-30 22:11:47.093318367 +0000 UTC m=+2301.567822347"
Oct 30 22:11:47.188094 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:11:47.188015    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:11:47.205210 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:47.205149    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c2fdb-jrs24" podStartSLOduration=4.205131105 podStartE2EDuration="4.205131105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:47.203862085 +0000 UTC m=+2301.678366064" watchObservedRunningTime="2026-10-30 22:11:47.205131105 +0000 UTC m=+2301.679635087"
Oct 30 22:11:47.711430 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:47.711256    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14ebb851-d1f0-43a8-8351-9fbe3e5a4e71" path="/var/lib/kubelet/pods/14ebb851-d1f0-43a8-8351-9fbe3e5a4e71/volumes"
Oct 30 22:11:47.716325 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:47.711772    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88a9407e-afe4-4e0b-af28-20226fb487c0" path="/var/lib/kubelet/pods/88a9407e-afe4-4e0b-af28-20226fb487c0/volumes"
Oct 30 22:11:47.993379 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:47.991300    5087 generic.go:356] "Generic (PLEG): container finished" podID="36ffceb8-7a93-46f7-897d-4a3a72b2864f" containerID="afabac5215053f697d1ef8032d2ce9c99e1a08643d2b591c0d78ca58adf1bac4" exitCode=0
Oct 30 22:11:47.993379 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:47.991499    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c035ad-wl8nc" event={"ID":"36ffceb8-7a93-46f7-897d-4a3a72b2864f","Type":"ContainerDied","Data":"afabac5215053f697d1ef8032d2ce9c99e1a08643d2b591c0d78ca58adf1bac4"}
Oct 30 22:11:47.995335 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:47.994890    5087 generic.go:356] "Generic (PLEG): container finished" podID="40934d2d-712a-4070-935e-cbeb5120c309" containerID="8e17283b9c1288f08f02f1408f8f968f6503aa741e018c1e9820083646d8e5d2" exitCode=0
Oct 30 22:11:47.995335 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:47.995038    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c2fdb-jrs24" event={"ID":"40934d2d-712a-4070-935e-cbeb5120c309","Type":"ContainerDied","Data":"8e17283b9c1288f08f02f1408f8f968f6503aa741e018c1e9820083646d8e5d2"}
Oct 30 22:11:48.754368 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:48.748526    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-pptz4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:11:48.754368 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:48.748642    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4" podUID="55c06e30-cb56-47cb-ae02-fc44e936bc48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:48.754754 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:48.754728    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-44qk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:11:48.754908 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:48.754885    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9" podUID="076fae6b-35bc-408e-aec8-e45316a2e9fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:48.759429 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:48.755409    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-pwfjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:11:48.759429 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:48.759027    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh" podUID="9c6d8084-9482-438d-b5e5-3f36442423d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:11:48.829279 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:48.829244    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:48.830778 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:48.830755    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:48.831989 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:48.831953    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:11:48.854322 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:48.854293    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:48.855258 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:48.855184    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:48.856443 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:48.856420    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:11:48.964751 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:48.964721    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:48.965049 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:48.965029    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:48.965888 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:48.965871    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:11:49.311481 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:49.311452    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c2fdb-jrs24"
Oct 30 22:11:49.317275 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:49.317254    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c035ad-wl8nc"
Oct 30 22:11:49.367422 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:49.361326    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36ffceb8-7a93-46f7-897d-4a3a72b2864f-kube-api-access-44lvl\" (UniqueName: \"kubernetes.io/projected/36ffceb8-7a93-46f7-897d-4a3a72b2864f-kube-api-access-44lvl\") pod \"36ffceb8-7a93-46f7-897d-4a3a72b2864f\" (UID: \"36ffceb8-7a93-46f7-897d-4a3a72b2864f\") "
Oct 30 22:11:49.367422 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:49.361494    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40934d2d-712a-4070-935e-cbeb5120c309-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40934d2d-712a-4070-935e-cbeb5120c309-maas-api-openshift-ca\") pod \"40934d2d-712a-4070-935e-cbeb5120c309\" (UID: \"40934d2d-712a-4070-935e-cbeb5120c309\") "
Oct 30 22:11:49.367422 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:49.361543    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40934d2d-712a-4070-935e-cbeb5120c309-kube-api-access-2wx6b\" (UniqueName: \"kubernetes.io/projected/40934d2d-712a-4070-935e-cbeb5120c309-kube-api-access-2wx6b\") pod \"40934d2d-712a-4070-935e-cbeb5120c309\" (UID: \"40934d2d-712a-4070-935e-cbeb5120c309\") "
Oct 30 22:11:49.367422 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:49.361581    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40934d2d-712a-4070-935e-cbeb5120c309-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40934d2d-712a-4070-935e-cbeb5120c309-maas-api-xks-ca\") pod \"40934d2d-712a-4070-935e-cbeb5120c309\" (UID: \"40934d2d-712a-4070-935e-cbeb5120c309\") "
Oct 30 22:11:49.367422 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:49.361615    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36ffceb8-7a93-46f7-897d-4a3a72b2864f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ffceb8-7a93-46f7-897d-4a3a72b2864f-maas-api-xks-ca\") pod \"36ffceb8-7a93-46f7-897d-4a3a72b2864f\" (UID: \"36ffceb8-7a93-46f7-897d-4a3a72b2864f\") "
Oct 30 22:11:49.367422 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:49.361660    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36ffceb8-7a93-46f7-897d-4a3a72b2864f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ffceb8-7a93-46f7-897d-4a3a72b2864f-maas-api-openshift-ca\") pod \"36ffceb8-7a93-46f7-897d-4a3a72b2864f\" (UID: \"36ffceb8-7a93-46f7-897d-4a3a72b2864f\") "
Oct 30 22:11:49.367422 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:49.361848    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40934d2d-712a-4070-935e-cbeb5120c309-maas-api-openshift-ca" pod "40934d2d-712a-4070-935e-cbeb5120c309" (UID: "40934d2d-712a-4070-935e-cbeb5120c309"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:49.367422 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:49.362193    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ffceb8-7a93-46f7-897d-4a3a72b2864f-maas-api-openshift-ca" pod "36ffceb8-7a93-46f7-897d-4a3a72b2864f" (UID: "36ffceb8-7a93-46f7-897d-4a3a72b2864f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:49.367422 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:49.362462    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40934d2d-712a-4070-935e-cbeb5120c309-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:49.367422 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:49.362479    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36ffceb8-7a93-46f7-897d-4a3a72b2864f-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:49.367422 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:49.364985    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40934d2d-712a-4070-935e-cbeb5120c309-kube-api-access-2wx6b" pod "40934d2d-712a-4070-935e-cbeb5120c309" (UID: "40934d2d-712a-4070-935e-cbeb5120c309"). InnerVolumeSpecName "kube-api-access-2wx6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:49.367422 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:49.366654    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36ffceb8-7a93-46f7-897d-4a3a72b2864f-maas-api-xks-ca" pod "36ffceb8-7a93-46f7-897d-4a3a72b2864f" (UID: "36ffceb8-7a93-46f7-897d-4a3a72b2864f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:49.367422 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:49.366912    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40934d2d-712a-4070-935e-cbeb5120c309-maas-api-xks-ca" pod "40934d2d-712a-4070-935e-cbeb5120c309" (UID: "40934d2d-712a-4070-935e-cbeb5120c309"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:11:49.373532 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:49.373491    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36ffceb8-7a93-46f7-897d-4a3a72b2864f-kube-api-access-44lvl" pod "36ffceb8-7a93-46f7-897d-4a3a72b2864f" (UID: "36ffceb8-7a93-46f7-897d-4a3a72b2864f"). InnerVolumeSpecName "kube-api-access-44lvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:49.463846 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:49.463804    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wx6b\" (UniqueName: \"kubernetes.io/projected/40934d2d-712a-4070-935e-cbeb5120c309-kube-api-access-2wx6b\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:49.463846 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:49.463848    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40934d2d-712a-4070-935e-cbeb5120c309-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:49.464100 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:49.463863    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36ffceb8-7a93-46f7-897d-4a3a72b2864f-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:49.464100 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:49.463876    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44lvl\" (UniqueName: \"kubernetes.io/projected/36ffceb8-7a93-46f7-897d-4a3a72b2864f-kube-api-access-44lvl\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:11:50.011400 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:50.011325    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c035ad-wl8nc" event={"ID":"36ffceb8-7a93-46f7-897d-4a3a72b2864f","Type":"ContainerDied","Data":"571c04f4351c2528bdc687959dd2726872ce6b91b96680e3913461284d6f5654"}
Oct 30 22:11:50.011653 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:50.011419    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="571c04f4351c2528bdc687959dd2726872ce6b91b96680e3913461284d6f5654"
Oct 30 22:11:50.011653 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:50.011472    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82c035ad-wl8nc"
Oct 30 22:11:50.013788 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:50.013765    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c2fdb-jrs24"
Oct 30 22:11:50.013949 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:50.013791    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f87c2fdb-jrs24" event={"ID":"40934d2d-712a-4070-935e-cbeb5120c309","Type":"ContainerDied","Data":"702127f52d027ed8455d9a34221ecad851989911dc8cd53638d23f5d8621dd54"}
Oct 30 22:11:50.013949 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:50.013839    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="702127f52d027ed8455d9a34221ecad851989911dc8cd53638d23f5d8621dd54"
Oct 30 22:11:52.326166 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:11:52.326132    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh" podUID="ad32b3aa-9229-422b-b7d5-6d2bd8531138" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:12:02.324471 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:02.324424    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh" podUID="ad32b3aa-9229-422b-b7d5-6d2bd8531138" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:12:03.028190 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:03.028164    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh"
Oct 30 22:12:03.145362 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:03.145306    5087 generic.go:356] "Generic (PLEG): container finished" podID="ad32b3aa-9229-422b-b7d5-6d2bd8531138" containerID="6eb8804136865c7eba0d9d31fedceca7c9ce278972d137ea5c02656a9a093f36" exitCode=0
Oct 30 22:12:03.145777 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:03.145745    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh" event={"ID":"ad32b3aa-9229-422b-b7d5-6d2bd8531138","Type":"ContainerDied","Data":"6eb8804136865c7eba0d9d31fedceca7c9ce278972d137ea5c02656a9a093f36"}
Oct 30 22:12:03.145917 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:03.145841    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh" event={"ID":"ad32b3aa-9229-422b-b7d5-6d2bd8531138","Type":"ContainerDied","Data":"77352ab3ee8fe4d25188be6b1a223bc346344e92f57234401a97191ee47d5ffd"}
Oct 30 22:12:03.145917 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:03.145871    5087 scope.go:122] "RemoveContainer" containerID="6eb8804136865c7eba0d9d31fedceca7c9ce278972d137ea5c02656a9a093f36"
Oct 30 22:12:03.146054 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:03.146040    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh"
Oct 30 22:12:03.153678 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:03.153648    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad32b3aa-9229-422b-b7d5-6d2bd8531138-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad32b3aa-9229-422b-b7d5-6d2bd8531138-plugins-config-volume\") pod \"ad32b3aa-9229-422b-b7d5-6d2bd8531138\" (UID: \"ad32b3aa-9229-422b-b7d5-6d2bd8531138\") "
Oct 30 22:12:03.153808 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:03.153766    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad32b3aa-9229-422b-b7d5-6d2bd8531138-kube-api-access-gf529\" (UniqueName: \"kubernetes.io/projected/ad32b3aa-9229-422b-b7d5-6d2bd8531138-kube-api-access-gf529\") pod \"ad32b3aa-9229-422b-b7d5-6d2bd8531138\" (UID: \"ad32b3aa-9229-422b-b7d5-6d2bd8531138\") "
Oct 30 22:12:03.154922 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:03.154887    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad32b3aa-9229-422b-b7d5-6d2bd8531138-plugins-config-volume" pod "ad32b3aa-9229-422b-b7d5-6d2bd8531138" (UID: "ad32b3aa-9229-422b-b7d5-6d2bd8531138"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:03.158484 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:03.158456    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad32b3aa-9229-422b-b7d5-6d2bd8531138-kube-api-access-gf529" pod "ad32b3aa-9229-422b-b7d5-6d2bd8531138" (UID: "ad32b3aa-9229-422b-b7d5-6d2bd8531138"). InnerVolumeSpecName "kube-api-access-gf529". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:03.161718 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:03.161617    5087 scope.go:122] "RemoveContainer" containerID="6eb8804136865c7eba0d9d31fedceca7c9ce278972d137ea5c02656a9a093f36"
Oct 30 22:12:03.162276 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:12:03.161937    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eb8804136865c7eba0d9d31fedceca7c9ce278972d137ea5c02656a9a093f36\": container with ID starting with 6eb8804136865c7eba0d9d31fedceca7c9ce278972d137ea5c02656a9a093f36 not found: ID does not exist" containerID="6eb8804136865c7eba0d9d31fedceca7c9ce278972d137ea5c02656a9a093f36"
Oct 30 22:12:03.162276 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:03.161972    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eb8804136865c7eba0d9d31fedceca7c9ce278972d137ea5c02656a9a093f36"} err="failed to get container status \"6eb8804136865c7eba0d9d31fedceca7c9ce278972d137ea5c02656a9a093f36\": rpc error: code = NotFound desc = could not find container \"6eb8804136865c7eba0d9d31fedceca7c9ce278972d137ea5c02656a9a093f36\": container with ID starting with 6eb8804136865c7eba0d9d31fedceca7c9ce278972d137ea5c02656a9a093f36 not found: ID does not exist"
Oct 30 22:12:03.255763 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:03.255675    5087 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad32b3aa-9229-422b-b7d5-6d2bd8531138-plugins-config-volume\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:03.255763 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:03.255702    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf529\" (UniqueName: \"kubernetes.io/projected/ad32b3aa-9229-422b-b7d5-6d2bd8531138-kube-api-access-gf529\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:03.573902 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:03.573805    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh"]
Oct 30 22:12:03.620727 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:03.620700    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-9b51e1-65b8fbbc9f-ng2zh"]
Oct 30 22:12:03.709101 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:03.709069    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad32b3aa-9229-422b-b7d5-6d2bd8531138" path="/var/lib/kubelet/pods/ad32b3aa-9229-422b-b7d5-6d2bd8531138/volumes"
Oct 30 22:12:09.125372 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:09.121475    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:12:09.416531 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:09.416421    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:12:09.437498 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:09.437446    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.89:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:12:09.544561 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:09.543436    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:12:09.573250 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:09.572432    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml" podUID="01fdf382-6833-463a-bed9-c7c096166819" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:12:09.643496 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:09.643434    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:12:09.721025 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:09.720821    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:12:09.785183 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:09.785141    5087 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/health\": context deadline exceeded"
Oct 30 22:12:16.413393 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:16.413358    5087 generic.go:356] "Generic (PLEG): container finished" podID="a6676e8e-cec6-431d-8a80-ae56c9843093" containerID="f767459022cd88fdbd9239afc6d4af710065bab5b3b1a5871a19dfb45e388d3c" exitCode=0
Oct 30 22:12:16.413929 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:16.413440    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42" event={"ID":"a6676e8e-cec6-431d-8a80-ae56c9843093","Type":"ContainerDied","Data":"f767459022cd88fdbd9239afc6d4af710065bab5b3b1a5871a19dfb45e388d3c"}
Oct 30 22:12:16.413929 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:16.413910    5087 scope.go:122] "RemoveContainer" containerID="f767459022cd88fdbd9239afc6d4af710065bab5b3b1a5871a19dfb45e388d3c"
Oct 30 22:12:16.415715 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:16.415686    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:12:16.415820 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:16.415766    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:12:16.415820 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:16.415811    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:12:17.193729 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:12:17.193695    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:12:17.425139 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:17.425109    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42" event={"ID":"a6676e8e-cec6-431d-8a80-ae56c9843093","Type":"ContainerStarted","Data":"988a607d076aed36fc14b181972df1c4d2e8d0af44ccc5fada3b8360c91d0b70"}
Oct 30 22:12:18.229096 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:18.229059    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:12:18.229274 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:18.229212    5087 patch_prober.go:29] interesting pod/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 22:12:18.229323 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:18.229268    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42" podUID="a6676e8e-cec6-431d-8a80-ae56c9843093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 22:12:19.229205 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:19.229168    5087 patch_prober.go:29] interesting pod/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 22:12:19.229640 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:19.229231    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42" podUID="a6676e8e-cec6-431d-8a80-ae56c9843093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 22:12:20.228671 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:20.228632    5087 patch_prober.go:29] interesting pod/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 22:12:20.228858 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:20.228697    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42" podUID="a6676e8e-cec6-431d-8a80-ae56c9843093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 22:12:21.228519 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:21.228485    5087 patch_prober.go:29] interesting pod/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 22:12:21.229014 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:21.228538    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42" podUID="a6676e8e-cec6-431d-8a80-ae56c9843093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 22:12:22.229379 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:22.229322    5087 patch_prober.go:29] interesting pod/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 30 22:12:22.229799 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:22.229408    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42" podUID="a6676e8e-cec6-431d-8a80-ae56c9843093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 30 22:12:23.233807 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:23.233774    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:12:23.234314 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:23.234144    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:12:23.234939 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:23.234918    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:12:27.781958 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.781920    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af27e8d0-zrk77"]
Oct 30 22:12:27.782705 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.782685    5087 cpu_manager.go:414] "containerMap: removing container" podUID="ad32b3aa-9229-422b-b7d5-6d2bd8531138" containerName="payload-processing"
Oct 30 22:12:27.782819 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.782706    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad32b3aa-9229-422b-b7d5-6d2bd8531138" containerName="payload-processing"
Oct 30 22:12:27.782819 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.782726    5087 cpu_manager.go:414] "containerMap: removing container" podUID="40934d2d-712a-4070-935e-cbeb5120c309" containerName="revoke-keys"
Oct 30 22:12:27.782819 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.782733    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40934d2d-712a-4070-935e-cbeb5120c309" containerName="revoke-keys"
Oct 30 22:12:27.782819 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.782747    5087 cpu_manager.go:414] "containerMap: removing container" podUID="36ffceb8-7a93-46f7-897d-4a3a72b2864f" containerName="revoke-keys"
Oct 30 22:12:27.782819 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.782754    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36ffceb8-7a93-46f7-897d-4a3a72b2864f" containerName="revoke-keys"
Oct 30 22:12:27.782819 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.782786    5087 cpu_manager.go:414] "containerMap: removing container" podUID="88a9407e-afe4-4e0b-af28-20226fb487c0" containerName="payload-pre-processing"
Oct 30 22:12:27.782819 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.782793    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88a9407e-afe4-4e0b-af28-20226fb487c0" containerName="payload-pre-processing"
Oct 30 22:12:27.782819 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.782816    5087 cpu_manager.go:414] "containerMap: removing container" podUID="14ebb851-d1f0-43a8-8351-9fbe3e5a4e71" containerName="payload-processing"
Oct 30 22:12:27.783232 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.782823    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14ebb851-d1f0-43a8-8351-9fbe3e5a4e71" containerName="payload-processing"
Oct 30 22:12:27.783232 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.782981    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad32b3aa-9229-422b-b7d5-6d2bd8531138" containerName="payload-processing"
Oct 30 22:12:27.783232 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.782997    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="40934d2d-712a-4070-935e-cbeb5120c309" containerName="revoke-keys"
Oct 30 22:12:27.783232 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.783006    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="36ffceb8-7a93-46f7-897d-4a3a72b2864f" containerName="revoke-keys"
Oct 30 22:12:27.783232 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.783018    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="88a9407e-afe4-4e0b-af28-20226fb487c0" containerName="payload-pre-processing"
Oct 30 22:12:27.783232 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.783031    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="14ebb851-d1f0-43a8-8351-9fbe3e5a4e71" containerName="payload-processing"
Oct 30 22:12:27.786720 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.786697    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af27e8d0-zrk77"
Oct 30 22:12:27.866004 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.865972    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96a18860-c4dc-4684-bccd-e0674098bdc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af27e8d0-zrk77\" (UID: \"96a18860-c4dc-4684-bccd-e0674098bdc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af27e8d0-zrk77"
Oct 30 22:12:27.866408 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.866376    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96a18860-c4dc-4684-bccd-e0674098bdc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af27e8d0-zrk77\" (UID: \"96a18860-c4dc-4684-bccd-e0674098bdc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af27e8d0-zrk77"
Oct 30 22:12:27.866548 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.866525    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc8kp\" (UniqueName: \"kubernetes.io/projected/96a18860-c4dc-4684-bccd-e0674098bdc5-kube-api-access-bc8kp\") pod \"maas-api-revoke-sub-af27e8d0-zrk77\" (UID: \"96a18860-c4dc-4684-bccd-e0674098bdc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af27e8d0-zrk77"
Oct 30 22:12:27.874471 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.874438    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:12:27.874605 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.874524    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:12:27.877786 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.877764    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-105770a5-v79d2"]
Oct 30 22:12:27.881995 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.881980    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105770a5-v79d2"
Oct 30 22:12:27.882075 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.882036    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af27e8d0-zrk77"]
Oct 30 22:12:27.937710 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.937678    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-105770a5-v79d2"]
Oct 30 22:12:27.967942 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.967911    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96a18860-c4dc-4684-bccd-e0674098bdc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af27e8d0-zrk77\" (UID: \"96a18860-c4dc-4684-bccd-e0674098bdc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af27e8d0-zrk77"
Oct 30 22:12:27.968134 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.967983    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc8kp\" (UniqueName: \"kubernetes.io/projected/96a18860-c4dc-4684-bccd-e0674098bdc5-kube-api-access-bc8kp\") pod \"maas-api-revoke-sub-af27e8d0-zrk77\" (UID: \"96a18860-c4dc-4684-bccd-e0674098bdc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af27e8d0-zrk77"
Oct 30 22:12:27.968134 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.968043    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wfpw\" (UniqueName: \"kubernetes.io/projected/724f88e6-a650-46a6-a26b-a812b30d6112-kube-api-access-2wfpw\") pod \"maas-api-revoke-sub-105770a5-v79d2\" (UID: \"724f88e6-a650-46a6-a26b-a812b30d6112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105770a5-v79d2"
Oct 30 22:12:27.968134 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.968063    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724f88e6-a650-46a6-a26b-a812b30d6112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-105770a5-v79d2\" (UID: \"724f88e6-a650-46a6-a26b-a812b30d6112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105770a5-v79d2"
Oct 30 22:12:27.968134 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.968088    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96a18860-c4dc-4684-bccd-e0674098bdc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af27e8d0-zrk77\" (UID: \"96a18860-c4dc-4684-bccd-e0674098bdc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af27e8d0-zrk77"
Oct 30 22:12:27.968134 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.968123    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724f88e6-a650-46a6-a26b-a812b30d6112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-105770a5-v79d2\" (UID: \"724f88e6-a650-46a6-a26b-a812b30d6112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105770a5-v79d2"
Oct 30 22:12:27.968952 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.968923    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96a18860-c4dc-4684-bccd-e0674098bdc5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af27e8d0-zrk77\" (UID: \"96a18860-c4dc-4684-bccd-e0674098bdc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af27e8d0-zrk77"
Oct 30 22:12:27.971151 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:27.971124    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96a18860-c4dc-4684-bccd-e0674098bdc5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af27e8d0-zrk77\" (UID: \"96a18860-c4dc-4684-bccd-e0674098bdc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af27e8d0-zrk77"
Oct 30 22:12:28.034865 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:28.034786    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc8kp\" (UniqueName: \"kubernetes.io/projected/96a18860-c4dc-4684-bccd-e0674098bdc5-kube-api-access-bc8kp\") pod \"maas-api-revoke-sub-af27e8d0-zrk77\" (UID: \"96a18860-c4dc-4684-bccd-e0674098bdc5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af27e8d0-zrk77"
Oct 30 22:12:28.069392 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:28.069330    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wfpw\" (UniqueName: \"kubernetes.io/projected/724f88e6-a650-46a6-a26b-a812b30d6112-kube-api-access-2wfpw\") pod \"maas-api-revoke-sub-105770a5-v79d2\" (UID: \"724f88e6-a650-46a6-a26b-a812b30d6112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105770a5-v79d2"
Oct 30 22:12:28.069568 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:28.069407    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724f88e6-a650-46a6-a26b-a812b30d6112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-105770a5-v79d2\" (UID: \"724f88e6-a650-46a6-a26b-a812b30d6112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105770a5-v79d2"
Oct 30 22:12:28.069568 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:28.069472    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724f88e6-a650-46a6-a26b-a812b30d6112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-105770a5-v79d2\" (UID: \"724f88e6-a650-46a6-a26b-a812b30d6112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105770a5-v79d2"
Oct 30 22:12:28.070161 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:28.070129    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724f88e6-a650-46a6-a26b-a812b30d6112-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-105770a5-v79d2\" (UID: \"724f88e6-a650-46a6-a26b-a812b30d6112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105770a5-v79d2"
Oct 30 22:12:28.072613 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:28.072588    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724f88e6-a650-46a6-a26b-a812b30d6112-maas-api-xks-ca\") pod \"maas-api-revoke-sub-105770a5-v79d2\" (UID: \"724f88e6-a650-46a6-a26b-a812b30d6112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105770a5-v79d2"
Oct 30 22:12:28.100468 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:28.100448    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af27e8d0-zrk77"
Oct 30 22:12:28.129500 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:28.129480    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wfpw\" (UniqueName: \"kubernetes.io/projected/724f88e6-a650-46a6-a26b-a812b30d6112-kube-api-access-2wfpw\") pod \"maas-api-revoke-sub-105770a5-v79d2\" (UID: \"724f88e6-a650-46a6-a26b-a812b30d6112\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-105770a5-v79d2"
Oct 30 22:12:28.197610 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:28.197578    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105770a5-v79d2"
Oct 30 22:12:28.513981 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:28.513948    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af27e8d0-zrk77"]
Oct 30 22:12:28.515818 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:12:28.515791    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a18860_c4dc_4684_bccd_e0674098bdc5.slice/crio-6e3070de87c56c4efd27ca41ff70e7cdf57d96b4cf5c624f389dade403b9a18d WatchSource:0}: Error finding container 6e3070de87c56c4efd27ca41ff70e7cdf57d96b4cf5c624f389dade403b9a18d: Status 404 returned error can't find the container with id 6e3070de87c56c4efd27ca41ff70e7cdf57d96b4cf5c624f389dade403b9a18d
Oct 30 22:12:28.536972 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:28.536945    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af27e8d0-zrk77" event={"ID":"96a18860-c4dc-4684-bccd-e0674098bdc5","Type":"ContainerStarted","Data":"6e3070de87c56c4efd27ca41ff70e7cdf57d96b4cf5c624f389dade403b9a18d"}
Oct 30 22:12:28.616977 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:12:28.616949    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod724f88e6_a650_46a6_a26b_a812b30d6112.slice/crio-9d656d710d36bef56253b47ea03e93a931ba42b07f45172075b515261318947e WatchSource:0}: Error finding container 9d656d710d36bef56253b47ea03e93a931ba42b07f45172075b515261318947e: Status 404 returned error can't find the container with id 9d656d710d36bef56253b47ea03e93a931ba42b07f45172075b515261318947e
Oct 30 22:12:28.617715 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:28.617634    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-105770a5-v79d2"]
Oct 30 22:12:29.546557 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:29.546524    5087 generic.go:356] "Generic (PLEG): container finished" podID="96a18860-c4dc-4684-bccd-e0674098bdc5" containerID="53786d0863d7e55572ae7dc6870afe201984938bf5b187654c121d704ad97a26" exitCode=0
Oct 30 22:12:29.547208 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:29.546606    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af27e8d0-zrk77" event={"ID":"96a18860-c4dc-4684-bccd-e0674098bdc5","Type":"ContainerDied","Data":"53786d0863d7e55572ae7dc6870afe201984938bf5b187654c121d704ad97a26"}
Oct 30 22:12:29.548657 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:29.548628    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105770a5-v79d2" event={"ID":"724f88e6-a650-46a6-a26b-a812b30d6112","Type":"ContainerStarted","Data":"9999eb380c55205e07a5f8fdb9e99ce521bc765973bc430325c99cab4877a9ca"}
Oct 30 22:12:29.548800 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:29.548683    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105770a5-v79d2" event={"ID":"724f88e6-a650-46a6-a26b-a812b30d6112","Type":"ContainerStarted","Data":"9d656d710d36bef56253b47ea03e93a931ba42b07f45172075b515261318947e"}
Oct 30 22:12:29.782472 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:29.782327    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105770a5-v79d2" podStartSLOduration=2.782311841 podStartE2EDuration="2.782311841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:29.781042983 +0000 UTC m=+2344.255546964" watchObservedRunningTime="2026-10-30 22:12:29.782311841 +0000 UTC m=+2344.256815821"
Oct 30 22:12:30.139903 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:30.139815    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v"]
Oct 30 22:12:30.142900 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:30.142869    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9f8e92-4jb4v"]
Oct 30 22:12:30.557913 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:30.557881    5087 generic.go:356] "Generic (PLEG): container finished" podID="724f88e6-a650-46a6-a26b-a812b30d6112" containerID="9999eb380c55205e07a5f8fdb9e99ce521bc765973bc430325c99cab4877a9ca" exitCode=0
Oct 30 22:12:30.558402 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:30.557971    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105770a5-v79d2" event={"ID":"724f88e6-a650-46a6-a26b-a812b30d6112","Type":"ContainerDied","Data":"9999eb380c55205e07a5f8fdb9e99ce521bc765973bc430325c99cab4877a9ca"}
Oct 30 22:12:30.718599 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:30.718572    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af27e8d0-zrk77"
Oct 30 22:12:30.824018 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:30.823925    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96a18860-c4dc-4684-bccd-e0674098bdc5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96a18860-c4dc-4684-bccd-e0674098bdc5-maas-api-openshift-ca\") pod \"96a18860-c4dc-4684-bccd-e0674098bdc5\" (UID: \"96a18860-c4dc-4684-bccd-e0674098bdc5\") "
Oct 30 22:12:30.824190 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:30.824131    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a18860-c4dc-4684-bccd-e0674098bdc5-kube-api-access-bc8kp\" (UniqueName: \"kubernetes.io/projected/96a18860-c4dc-4684-bccd-e0674098bdc5-kube-api-access-bc8kp\") pod \"96a18860-c4dc-4684-bccd-e0674098bdc5\" (UID: \"96a18860-c4dc-4684-bccd-e0674098bdc5\") "
Oct 30 22:12:30.824190 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:30.824172    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96a18860-c4dc-4684-bccd-e0674098bdc5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96a18860-c4dc-4684-bccd-e0674098bdc5-maas-api-xks-ca\") pod \"96a18860-c4dc-4684-bccd-e0674098bdc5\" (UID: \"96a18860-c4dc-4684-bccd-e0674098bdc5\") "
Oct 30 22:12:30.824308 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:30.824279    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a18860-c4dc-4684-bccd-e0674098bdc5-maas-api-openshift-ca" pod "96a18860-c4dc-4684-bccd-e0674098bdc5" (UID: "96a18860-c4dc-4684-bccd-e0674098bdc5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:30.824698 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:30.824678    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96a18860-c4dc-4684-bccd-e0674098bdc5-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:30.826400 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:30.826366    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96a18860-c4dc-4684-bccd-e0674098bdc5-maas-api-xks-ca" pod "96a18860-c4dc-4684-bccd-e0674098bdc5" (UID: "96a18860-c4dc-4684-bccd-e0674098bdc5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:30.826513 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:30.826401    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a18860-c4dc-4684-bccd-e0674098bdc5-kube-api-access-bc8kp" pod "96a18860-c4dc-4684-bccd-e0674098bdc5" (UID: "96a18860-c4dc-4684-bccd-e0674098bdc5"). InnerVolumeSpecName "kube-api-access-bc8kp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:30.925962 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:30.925926    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc8kp\" (UniqueName: \"kubernetes.io/projected/96a18860-c4dc-4684-bccd-e0674098bdc5-kube-api-access-bc8kp\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:30.925962 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:30.925957    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96a18860-c4dc-4684-bccd-e0674098bdc5-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:31.567665 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:31.567625    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af27e8d0-zrk77" event={"ID":"96a18860-c4dc-4684-bccd-e0674098bdc5","Type":"ContainerDied","Data":"6e3070de87c56c4efd27ca41ff70e7cdf57d96b4cf5c624f389dade403b9a18d"}
Oct 30 22:12:31.568173 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:31.567644    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af27e8d0-zrk77"
Oct 30 22:12:31.568173 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:31.567724    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e3070de87c56c4efd27ca41ff70e7cdf57d96b4cf5c624f389dade403b9a18d"
Oct 30 22:12:31.711547 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:31.711517    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f847f946-690e-4d4c-a6b4-3585f52b31db" path="/var/lib/kubelet/pods/f847f946-690e-4d4c-a6b4-3585f52b31db/volumes"
Oct 30 22:12:31.718003 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:31.717982    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105770a5-v79d2"
Oct 30 22:12:31.842106 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:31.841107    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/724f88e6-a650-46a6-a26b-a812b30d6112-kube-api-access-2wfpw\" (UniqueName: \"kubernetes.io/projected/724f88e6-a650-46a6-a26b-a812b30d6112-kube-api-access-2wfpw\") pod \"724f88e6-a650-46a6-a26b-a812b30d6112\" (UID: \"724f88e6-a650-46a6-a26b-a812b30d6112\") "
Oct 30 22:12:31.842106 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:31.841213    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/724f88e6-a650-46a6-a26b-a812b30d6112-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724f88e6-a650-46a6-a26b-a812b30d6112-maas-api-openshift-ca\") pod \"724f88e6-a650-46a6-a26b-a812b30d6112\" (UID: \"724f88e6-a650-46a6-a26b-a812b30d6112\") "
Oct 30 22:12:31.842106 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:31.841291    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/724f88e6-a650-46a6-a26b-a812b30d6112-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724f88e6-a650-46a6-a26b-a812b30d6112-maas-api-xks-ca\") pod \"724f88e6-a650-46a6-a26b-a812b30d6112\" (UID: \"724f88e6-a650-46a6-a26b-a812b30d6112\") "
Oct 30 22:12:31.846128 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:31.845897    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/724f88e6-a650-46a6-a26b-a812b30d6112-kube-api-access-2wfpw" pod "724f88e6-a650-46a6-a26b-a812b30d6112" (UID: "724f88e6-a650-46a6-a26b-a812b30d6112"). InnerVolumeSpecName "kube-api-access-2wfpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:31.846733 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:31.846468    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/724f88e6-a650-46a6-a26b-a812b30d6112-maas-api-openshift-ca" pod "724f88e6-a650-46a6-a26b-a812b30d6112" (UID: "724f88e6-a650-46a6-a26b-a812b30d6112"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:31.874514 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:31.874480    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/724f88e6-a650-46a6-a26b-a812b30d6112-maas-api-xks-ca" pod "724f88e6-a650-46a6-a26b-a812b30d6112" (UID: "724f88e6-a650-46a6-a26b-a812b30d6112"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:31.943653 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:31.943621    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wfpw\" (UniqueName: \"kubernetes.io/projected/724f88e6-a650-46a6-a26b-a812b30d6112-kube-api-access-2wfpw\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:31.943653 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:31.943648    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/724f88e6-a650-46a6-a26b-a812b30d6112-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:31.943653 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:31.943657    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/724f88e6-a650-46a6-a26b-a812b30d6112-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:32.576048 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:32.576018    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105770a5-v79d2"
Oct 30 22:12:32.576554 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:32.576044    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-105770a5-v79d2" event={"ID":"724f88e6-a650-46a6-a26b-a812b30d6112","Type":"ContainerDied","Data":"9d656d710d36bef56253b47ea03e93a931ba42b07f45172075b515261318947e"}
Oct 30 22:12:32.576554 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:32.576118    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d656d710d36bef56253b47ea03e93a931ba42b07f45172075b515261318947e"
Oct 30 22:12:38.001194 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:38.001161    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-307368-xcfts"]
Oct 30 22:12:38.001811 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:38.001793    5087 cpu_manager.go:414] "containerMap: removing container" podUID="724f88e6-a650-46a6-a26b-a812b30d6112" containerName="revoke-keys"
Oct 30 22:12:38.001853 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:38.001813    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="724f88e6-a650-46a6-a26b-a812b30d6112" containerName="revoke-keys"
Oct 30 22:12:38.001889 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:38.001856    5087 cpu_manager.go:414] "containerMap: removing container" podUID="96a18860-c4dc-4684-bccd-e0674098bdc5" containerName="revoke-keys"
Oct 30 22:12:38.001889 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:38.001861    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a18860-c4dc-4684-bccd-e0674098bdc5" containerName="revoke-keys"
Oct 30 22:12:38.001974 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:38.001941    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="724f88e6-a650-46a6-a26b-a812b30d6112" containerName="revoke-keys"
Oct 30 22:12:38.001974 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:38.001959    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="96a18860-c4dc-4684-bccd-e0674098bdc5" containerName="revoke-keys"
Oct 30 22:12:38.005643 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:38.005626    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-307368-xcfts"
Oct 30 22:12:38.109320 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:38.109283    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:12:38.109595 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:38.109295    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:12:38.112761 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:38.112734    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-307368-xcfts"]
Oct 30 22:12:38.135114 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:38.135076    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b82f951-797b-437e-94c8-4892364f426f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-307368-xcfts\" (UID: \"6b82f951-797b-437e-94c8-4892364f426f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-307368-xcfts"
Oct 30 22:12:38.135291 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:38.135274    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b82f951-797b-437e-94c8-4892364f426f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-307368-xcfts\" (UID: \"6b82f951-797b-437e-94c8-4892364f426f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-307368-xcfts"
Oct 30 22:12:38.135402 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:38.135361    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdhq7\" (UniqueName: \"kubernetes.io/projected/6b82f951-797b-437e-94c8-4892364f426f-kube-api-access-wdhq7\") pod \"maas-api-revoke-keys-e2e-api-b-307368-xcfts\" (UID: \"6b82f951-797b-437e-94c8-4892364f426f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-307368-xcfts"
Oct 30 22:12:38.236285 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:38.236248    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b82f951-797b-437e-94c8-4892364f426f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-307368-xcfts\" (UID: \"6b82f951-797b-437e-94c8-4892364f426f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-307368-xcfts"
Oct 30 22:12:38.236501 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:38.236322    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdhq7\" (UniqueName: \"kubernetes.io/projected/6b82f951-797b-437e-94c8-4892364f426f-kube-api-access-wdhq7\") pod \"maas-api-revoke-keys-e2e-api-b-307368-xcfts\" (UID: \"6b82f951-797b-437e-94c8-4892364f426f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-307368-xcfts"
Oct 30 22:12:38.236501 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:38.236448    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b82f951-797b-437e-94c8-4892364f426f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-307368-xcfts\" (UID: \"6b82f951-797b-437e-94c8-4892364f426f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-307368-xcfts"
Oct 30 22:12:38.237141 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:38.237112    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b82f951-797b-437e-94c8-4892364f426f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-307368-xcfts\" (UID: \"6b82f951-797b-437e-94c8-4892364f426f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-307368-xcfts"
Oct 30 22:12:38.239697 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:38.239671    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b82f951-797b-437e-94c8-4892364f426f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-307368-xcfts\" (UID: \"6b82f951-797b-437e-94c8-4892364f426f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-307368-xcfts"
Oct 30 22:12:38.295652 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:38.295585    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdhq7\" (UniqueName: \"kubernetes.io/projected/6b82f951-797b-437e-94c8-4892364f426f-kube-api-access-wdhq7\") pod \"maas-api-revoke-keys-e2e-api-b-307368-xcfts\" (UID: \"6b82f951-797b-437e-94c8-4892364f426f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-307368-xcfts"
Oct 30 22:12:38.319487 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:38.319458    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-307368-xcfts"
Oct 30 22:12:38.712087 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:38.712052    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-307368-xcfts"]
Oct 30 22:12:38.713506 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:12:38.713473    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b82f951_797b_437e_94c8_4892364f426f.slice/crio-c93e63a405b62847cd333eaeecf43f7dee3ff16c37039a929563bb21988eb75b WatchSource:0}: Error finding container c93e63a405b62847cd333eaeecf43f7dee3ff16c37039a929563bb21988eb75b: Status 404 returned error can't find the container with id c93e63a405b62847cd333eaeecf43f7dee3ff16c37039a929563bb21988eb75b
Oct 30 22:12:39.634183 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:39.634147    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-307368-xcfts" event={"ID":"6b82f951-797b-437e-94c8-4892364f426f","Type":"ContainerStarted","Data":"96020f9af74e9c23825fb0e96b3c6a87d867bcf5af41c4d3700720239a87c48d"}
Oct 30 22:12:39.634590 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:39.634204    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-307368-xcfts" event={"ID":"6b82f951-797b-437e-94c8-4892364f426f","Type":"ContainerStarted","Data":"c93e63a405b62847cd333eaeecf43f7dee3ff16c37039a929563bb21988eb75b"}
Oct 30 22:12:39.750152 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:39.750086    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-307368-xcfts" podStartSLOduration=2.750066442 podStartE2EDuration="2.750066442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:39.746775856 +0000 UTC m=+2354.221279858" watchObservedRunningTime="2026-10-30 22:12:39.750066442 +0000 UTC m=+2354.224570422"
Oct 30 22:12:40.644266 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:40.644224    5087 generic.go:356] "Generic (PLEG): container finished" podID="6b82f951-797b-437e-94c8-4892364f426f" containerID="96020f9af74e9c23825fb0e96b3c6a87d867bcf5af41c4d3700720239a87c48d" exitCode=0
Oct 30 22:12:40.644732 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:40.644465    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-307368-xcfts" event={"ID":"6b82f951-797b-437e-94c8-4892364f426f","Type":"ContainerDied","Data":"96020f9af74e9c23825fb0e96b3c6a87d867bcf5af41c4d3700720239a87c48d"}
Oct 30 22:12:41.808258 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:41.808235    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-307368-xcfts"
Oct 30 22:12:41.899742 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:41.899710    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b82f951-797b-437e-94c8-4892364f426f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b82f951-797b-437e-94c8-4892364f426f-maas-api-xks-ca\") pod \"6b82f951-797b-437e-94c8-4892364f426f\" (UID: \"6b82f951-797b-437e-94c8-4892364f426f\") "
Oct 30 22:12:41.899954 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:41.899760    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b82f951-797b-437e-94c8-4892364f426f-kube-api-access-wdhq7\" (UniqueName: \"kubernetes.io/projected/6b82f951-797b-437e-94c8-4892364f426f-kube-api-access-wdhq7\") pod \"6b82f951-797b-437e-94c8-4892364f426f\" (UID: \"6b82f951-797b-437e-94c8-4892364f426f\") "
Oct 30 22:12:41.899954 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:41.899836    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b82f951-797b-437e-94c8-4892364f426f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b82f951-797b-437e-94c8-4892364f426f-maas-api-openshift-ca\") pod \"6b82f951-797b-437e-94c8-4892364f426f\" (UID: \"6b82f951-797b-437e-94c8-4892364f426f\") "
Oct 30 22:12:41.900187 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:41.900162    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b82f951-797b-437e-94c8-4892364f426f-maas-api-openshift-ca" pod "6b82f951-797b-437e-94c8-4892364f426f" (UID: "6b82f951-797b-437e-94c8-4892364f426f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:41.900519 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:41.900269    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b82f951-797b-437e-94c8-4892364f426f-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:41.902038 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:41.902011    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b82f951-797b-437e-94c8-4892364f426f-maas-api-xks-ca" pod "6b82f951-797b-437e-94c8-4892364f426f" (UID: "6b82f951-797b-437e-94c8-4892364f426f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:41.902197 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:41.902176    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b82f951-797b-437e-94c8-4892364f426f-kube-api-access-wdhq7" pod "6b82f951-797b-437e-94c8-4892364f426f" (UID: "6b82f951-797b-437e-94c8-4892364f426f"). InnerVolumeSpecName "kube-api-access-wdhq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:42.001030 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:42.000996    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b82f951-797b-437e-94c8-4892364f426f-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:42.001030 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:42.001028    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdhq7\" (UniqueName: \"kubernetes.io/projected/6b82f951-797b-437e-94c8-4892364f426f-kube-api-access-wdhq7\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:42.661912 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:42.661887    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-307368-xcfts"
Oct 30 22:12:42.662067 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:42.661922    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-307368-xcfts" event={"ID":"6b82f951-797b-437e-94c8-4892364f426f","Type":"ContainerDied","Data":"c93e63a405b62847cd333eaeecf43f7dee3ff16c37039a929563bb21988eb75b"}
Oct 30 22:12:42.662067 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:42.661971    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c93e63a405b62847cd333eaeecf43f7dee3ff16c37039a929563bb21988eb75b"
Oct 30 22:12:47.167522 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:12:47.167485    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:12:48.073192 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:48.073155    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dda9f5-v2pzf"]
Oct 30 22:12:48.120877 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:48.120847    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-dda9f5-v2pzf"]
Oct 30 22:12:48.476152 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:48.476080    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-307368-7574bb79f5-t8w22"]
Oct 30 22:12:48.582125 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:48.582088    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-307368-68678879f5-dtgqs"]
Oct 30 22:12:49.712983 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:49.712947    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1269e40-c1fe-4166-ad83-6f07a8c95169" path="/var/lib/kubelet/pods/a1269e40-c1fe-4166-ad83-6f07a8c95169/volumes"
Oct 30 22:12:51.769946 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:51.769904    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f84a2ad-bc4cc"]
Oct 30 22:12:51.771513 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:51.771477    5087 cpu_manager.go:414] "containerMap: removing container" podUID="6b82f951-797b-437e-94c8-4892364f426f" containerName="revoke-keys"
Oct 30 22:12:51.771513 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:51.771505    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b82f951-797b-437e-94c8-4892364f426f" containerName="revoke-keys"
Oct 30 22:12:51.771786 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:51.771651    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b82f951-797b-437e-94c8-4892364f426f" containerName="revoke-keys"
Oct 30 22:12:51.775357 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:51.775330    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f84a2ad-bc4cc"
Oct 30 22:12:51.831258 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:51.831224    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f84a2ad-bc4cc"]
Oct 30 22:12:51.845873 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:51.845842    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c30ec77-1b75-482e-a713-0aaff964756f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f84a2ad-bc4cc\" (UID: \"6c30ec77-1b75-482e-a713-0aaff964756f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f84a2ad-bc4cc"
Oct 30 22:12:51.846114 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:51.846092    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwxk2\" (UniqueName: \"kubernetes.io/projected/6c30ec77-1b75-482e-a713-0aaff964756f-kube-api-access-fwxk2\") pod \"maas-api-revoke-sub-4f84a2ad-bc4cc\" (UID: \"6c30ec77-1b75-482e-a713-0aaff964756f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f84a2ad-bc4cc"
Oct 30 22:12:51.846264 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:51.846248    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c30ec77-1b75-482e-a713-0aaff964756f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f84a2ad-bc4cc\" (UID: \"6c30ec77-1b75-482e-a713-0aaff964756f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f84a2ad-bc4cc"
Oct 30 22:12:51.876656 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:51.876630    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:12:51.876795 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:51.876681    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:12:51.948188 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:51.948155    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c30ec77-1b75-482e-a713-0aaff964756f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f84a2ad-bc4cc\" (UID: \"6c30ec77-1b75-482e-a713-0aaff964756f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f84a2ad-bc4cc"
Oct 30 22:12:51.948378 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:51.948204    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwxk2\" (UniqueName: \"kubernetes.io/projected/6c30ec77-1b75-482e-a713-0aaff964756f-kube-api-access-fwxk2\") pod \"maas-api-revoke-sub-4f84a2ad-bc4cc\" (UID: \"6c30ec77-1b75-482e-a713-0aaff964756f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f84a2ad-bc4cc"
Oct 30 22:12:51.948378 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:51.948238    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c30ec77-1b75-482e-a713-0aaff964756f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f84a2ad-bc4cc\" (UID: \"6c30ec77-1b75-482e-a713-0aaff964756f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f84a2ad-bc4cc"
Oct 30 22:12:51.948982 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:51.948956    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c30ec77-1b75-482e-a713-0aaff964756f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f84a2ad-bc4cc\" (UID: \"6c30ec77-1b75-482e-a713-0aaff964756f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f84a2ad-bc4cc"
Oct 30 22:12:51.950876 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:51.950850    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c30ec77-1b75-482e-a713-0aaff964756f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f84a2ad-bc4cc\" (UID: \"6c30ec77-1b75-482e-a713-0aaff964756f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f84a2ad-bc4cc"
Oct 30 22:12:52.004956 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:52.004927    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwxk2\" (UniqueName: \"kubernetes.io/projected/6c30ec77-1b75-482e-a713-0aaff964756f-kube-api-access-fwxk2\") pod \"maas-api-revoke-sub-4f84a2ad-bc4cc\" (UID: \"6c30ec77-1b75-482e-a713-0aaff964756f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f84a2ad-bc4cc"
Oct 30 22:12:52.092106 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:52.092036    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f84a2ad-bc4cc"
Oct 30 22:12:52.472703 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:52.472673    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f84a2ad-bc4cc"]
Oct 30 22:12:52.474539 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:12:52.474510    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c30ec77_1b75_482e_a713_0aaff964756f.slice/crio-1f48b3fa2c41e15bc92b7d38abd64b427fd9d09d4bbcdce57e09f8bc3d578b6b WatchSource:0}: Error finding container 1f48b3fa2c41e15bc92b7d38abd64b427fd9d09d4bbcdce57e09f8bc3d578b6b: Status 404 returned error can't find the container with id 1f48b3fa2c41e15bc92b7d38abd64b427fd9d09d4bbcdce57e09f8bc3d578b6b
Oct 30 22:12:52.767605 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:52.767574    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f84a2ad-bc4cc" event={"ID":"6c30ec77-1b75-482e-a713-0aaff964756f","Type":"ContainerStarted","Data":"046e43d9218239342990af0b656c22d9bee82c5c38d7a01a26c09b88386ac131"}
Oct 30 22:12:52.767767 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:52.767654    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f84a2ad-bc4cc" event={"ID":"6c30ec77-1b75-482e-a713-0aaff964756f","Type":"ContainerStarted","Data":"1f48b3fa2c41e15bc92b7d38abd64b427fd9d09d4bbcdce57e09f8bc3d578b6b"}
Oct 30 22:12:52.883053 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:52.883001    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f84a2ad-bc4cc" podStartSLOduration=1.882985538 podStartE2EDuration="1.882985538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:52.881401507 +0000 UTC m=+2367.355905487" watchObservedRunningTime="2026-10-30 22:12:52.882985538 +0000 UTC m=+2367.357489516"
Oct 30 22:12:53.476655 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:53.476596    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-307368-7574bb79f5-t8w22" podUID="dcb28650-f1aa-4f60-a062-bd7dc84418eb" containerName="payload-processing" containerID="cri-o://9442fd8b27cbf88cfce8755285a9248c0f23a451bf092adf71a5da647d3cc455" gracePeriod=25
Oct 30 22:12:53.582767 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:53.582719    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-307368-68678879f5-dtgqs" podUID="c96595e9-95b1-4a92-a7ad-93932a2001ba" containerName="payload-pre-processing" containerID="cri-o://43d95aae1dff34f4c912fbfdd01082bb7dd090c756f39fa349d72fb6fcbeaf80" gracePeriod=25
Oct 30 22:12:53.769308 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:53.769281    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-307368-7574bb79f5-t8w22"
Oct 30 22:12:53.781108 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:53.780940    5087 generic.go:356] "Generic (PLEG): container finished" podID="c96595e9-95b1-4a92-a7ad-93932a2001ba" containerID="43d95aae1dff34f4c912fbfdd01082bb7dd090c756f39fa349d72fb6fcbeaf80" exitCode=0
Oct 30 22:12:53.781195 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:53.781121    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-307368-68678879f5-dtgqs" event={"ID":"c96595e9-95b1-4a92-a7ad-93932a2001ba","Type":"ContainerDied","Data":"43d95aae1dff34f4c912fbfdd01082bb7dd090c756f39fa349d72fb6fcbeaf80"}
Oct 30 22:12:53.782924 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:53.782896    5087 generic.go:356] "Generic (PLEG): container finished" podID="dcb28650-f1aa-4f60-a062-bd7dc84418eb" containerID="9442fd8b27cbf88cfce8755285a9248c0f23a451bf092adf71a5da647d3cc455" exitCode=0
Oct 30 22:12:53.783063 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:53.783004    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-307368-7574bb79f5-t8w22" event={"ID":"dcb28650-f1aa-4f60-a062-bd7dc84418eb","Type":"ContainerDied","Data":"9442fd8b27cbf88cfce8755285a9248c0f23a451bf092adf71a5da647d3cc455"}
Oct 30 22:12:53.783063 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:53.783054    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-307368-7574bb79f5-t8w22" event={"ID":"dcb28650-f1aa-4f60-a062-bd7dc84418eb","Type":"ContainerDied","Data":"831668da4e94568b5d98347547ac880179435266b517713a58130b9e918ea8ee"}
Oct 30 22:12:53.783175 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:53.783084    5087 scope.go:122] "RemoveContainer" containerID="9442fd8b27cbf88cfce8755285a9248c0f23a451bf092adf71a5da647d3cc455"
Oct 30 22:12:53.783276 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:53.783261    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-307368-7574bb79f5-t8w22"
Oct 30 22:12:53.786636 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:53.786609    5087 generic.go:356] "Generic (PLEG): container finished" podID="6c30ec77-1b75-482e-a713-0aaff964756f" containerID="046e43d9218239342990af0b656c22d9bee82c5c38d7a01a26c09b88386ac131" exitCode=0
Oct 30 22:12:53.786749 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:53.786677    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f84a2ad-bc4cc" event={"ID":"6c30ec77-1b75-482e-a713-0aaff964756f","Type":"ContainerDied","Data":"046e43d9218239342990af0b656c22d9bee82c5c38d7a01a26c09b88386ac131"}
Oct 30 22:12:53.801289 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:53.801272    5087 scope.go:122] "RemoveContainer" containerID="9442fd8b27cbf88cfce8755285a9248c0f23a451bf092adf71a5da647d3cc455"
Oct 30 22:12:53.801647 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:12:53.801619    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9442fd8b27cbf88cfce8755285a9248c0f23a451bf092adf71a5da647d3cc455\": container with ID starting with 9442fd8b27cbf88cfce8755285a9248c0f23a451bf092adf71a5da647d3cc455 not found: ID does not exist" containerID="9442fd8b27cbf88cfce8755285a9248c0f23a451bf092adf71a5da647d3cc455"
Oct 30 22:12:53.801774 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:53.801656    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9442fd8b27cbf88cfce8755285a9248c0f23a451bf092adf71a5da647d3cc455"} err="failed to get container status \"9442fd8b27cbf88cfce8755285a9248c0f23a451bf092adf71a5da647d3cc455\": rpc error: code = NotFound desc = could not find container \"9442fd8b27cbf88cfce8755285a9248c0f23a451bf092adf71a5da647d3cc455\": container with ID starting with 9442fd8b27cbf88cfce8755285a9248c0f23a451bf092adf71a5da647d3cc455 not found: ID does not exist"
Oct 30 22:12:53.878445 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:53.878418    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcb28650-f1aa-4f60-a062-bd7dc84418eb-kube-api-access-mfw85\" (UniqueName: \"kubernetes.io/projected/dcb28650-f1aa-4f60-a062-bd7dc84418eb-kube-api-access-mfw85\") pod \"dcb28650-f1aa-4f60-a062-bd7dc84418eb\" (UID: \"dcb28650-f1aa-4f60-a062-bd7dc84418eb\") "
Oct 30 22:12:53.878604 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:53.878467    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcb28650-f1aa-4f60-a062-bd7dc84418eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb28650-f1aa-4f60-a062-bd7dc84418eb-plugins-config-volume\") pod \"dcb28650-f1aa-4f60-a062-bd7dc84418eb\" (UID: \"dcb28650-f1aa-4f60-a062-bd7dc84418eb\") "
Oct 30 22:12:53.878936 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:53.878909    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcb28650-f1aa-4f60-a062-bd7dc84418eb-plugins-config-volume" pod "dcb28650-f1aa-4f60-a062-bd7dc84418eb" (UID: "dcb28650-f1aa-4f60-a062-bd7dc84418eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:53.879372 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:53.879332    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-307368-68678879f5-dtgqs"
Oct 30 22:12:53.880798 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:53.880773    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcb28650-f1aa-4f60-a062-bd7dc84418eb-kube-api-access-mfw85" pod "dcb28650-f1aa-4f60-a062-bd7dc84418eb" (UID: "dcb28650-f1aa-4f60-a062-bd7dc84418eb"). InnerVolumeSpecName "kube-api-access-mfw85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:53.980146 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:53.980109    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c96595e9-95b1-4a92-a7ad-93932a2001ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c96595e9-95b1-4a92-a7ad-93932a2001ba-plugins-config-volume\") pod \"c96595e9-95b1-4a92-a7ad-93932a2001ba\" (UID: \"c96595e9-95b1-4a92-a7ad-93932a2001ba\") "
Oct 30 22:12:53.980528 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:53.980191    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c96595e9-95b1-4a92-a7ad-93932a2001ba-kube-api-access-xbw9f\" (UniqueName: \"kubernetes.io/projected/c96595e9-95b1-4a92-a7ad-93932a2001ba-kube-api-access-xbw9f\") pod \"c96595e9-95b1-4a92-a7ad-93932a2001ba\" (UID: \"c96595e9-95b1-4a92-a7ad-93932a2001ba\") "
Oct 30 22:12:53.980581 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:53.980550    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c96595e9-95b1-4a92-a7ad-93932a2001ba-plugins-config-volume" pod "c96595e9-95b1-4a92-a7ad-93932a2001ba" (UID: "c96595e9-95b1-4a92-a7ad-93932a2001ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:53.980947 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:53.980926    5087 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c96595e9-95b1-4a92-a7ad-93932a2001ba-plugins-config-volume\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:53.981019 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:53.980953    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfw85\" (UniqueName: \"kubernetes.io/projected/dcb28650-f1aa-4f60-a062-bd7dc84418eb-kube-api-access-mfw85\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:53.981019 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:53.980968    5087 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb28650-f1aa-4f60-a062-bd7dc84418eb-plugins-config-volume\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:53.982797 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:53.982774    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c96595e9-95b1-4a92-a7ad-93932a2001ba-kube-api-access-xbw9f" pod "c96595e9-95b1-4a92-a7ad-93932a2001ba" (UID: "c96595e9-95b1-4a92-a7ad-93932a2001ba"). InnerVolumeSpecName "kube-api-access-xbw9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:54.082914 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:54.082874    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbw9f\" (UniqueName: \"kubernetes.io/projected/c96595e9-95b1-4a92-a7ad-93932a2001ba-kube-api-access-xbw9f\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:54.307386 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:54.307325    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-307368-7574bb79f5-t8w22"]
Oct 30 22:12:54.310357 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:54.310322    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-307368-7574bb79f5-t8w22"]
Oct 30 22:12:54.796443 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:54.796417    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-307368-68678879f5-dtgqs"
Oct 30 22:12:54.796652 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:54.796452    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-307368-68678879f5-dtgqs" event={"ID":"c96595e9-95b1-4a92-a7ad-93932a2001ba","Type":"ContainerDied","Data":"14685ad41d17bdf79fdbaf255ddc028c869c408ad88b1b86b18ebf1327b04ae3"}
Oct 30 22:12:54.796652 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:54.796528    5087 scope.go:122] "RemoveContainer" containerID="43d95aae1dff34f4c912fbfdd01082bb7dd090c756f39fa349d72fb6fcbeaf80"
Oct 30 22:12:54.959401 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:54.959374    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f84a2ad-bc4cc"
Oct 30 22:12:54.966492 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:54.966464    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-307368-68678879f5-dtgqs"]
Oct 30 22:12:55.014637 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:55.014612    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-307368-68678879f5-dtgqs"]
Oct 30 22:12:55.098231 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:55.098154    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c30ec77-1b75-482e-a713-0aaff964756f-kube-api-access-fwxk2\" (UniqueName: \"kubernetes.io/projected/6c30ec77-1b75-482e-a713-0aaff964756f-kube-api-access-fwxk2\") pod \"6c30ec77-1b75-482e-a713-0aaff964756f\" (UID: \"6c30ec77-1b75-482e-a713-0aaff964756f\") "
Oct 30 22:12:55.098430 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:55.098240    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c30ec77-1b75-482e-a713-0aaff964756f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c30ec77-1b75-482e-a713-0aaff964756f-maas-api-openshift-ca\") pod \"6c30ec77-1b75-482e-a713-0aaff964756f\" (UID: \"6c30ec77-1b75-482e-a713-0aaff964756f\") "
Oct 30 22:12:55.098430 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:55.098415    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c30ec77-1b75-482e-a713-0aaff964756f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c30ec77-1b75-482e-a713-0aaff964756f-maas-api-xks-ca\") pod \"6c30ec77-1b75-482e-a713-0aaff964756f\" (UID: \"6c30ec77-1b75-482e-a713-0aaff964756f\") "
Oct 30 22:12:55.098693 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:55.098657    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c30ec77-1b75-482e-a713-0aaff964756f-maas-api-openshift-ca" pod "6c30ec77-1b75-482e-a713-0aaff964756f" (UID: "6c30ec77-1b75-482e-a713-0aaff964756f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:55.100579 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:55.100546    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c30ec77-1b75-482e-a713-0aaff964756f-maas-api-xks-ca" pod "6c30ec77-1b75-482e-a713-0aaff964756f" (UID: "6c30ec77-1b75-482e-a713-0aaff964756f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:55.100579 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:55.100566    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c30ec77-1b75-482e-a713-0aaff964756f-kube-api-access-fwxk2" pod "6c30ec77-1b75-482e-a713-0aaff964756f" (UID: "6c30ec77-1b75-482e-a713-0aaff964756f"). InnerVolumeSpecName "kube-api-access-fwxk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:55.200227 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:55.200191    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwxk2\" (UniqueName: \"kubernetes.io/projected/6c30ec77-1b75-482e-a713-0aaff964756f-kube-api-access-fwxk2\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:55.200227 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:55.200226    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c30ec77-1b75-482e-a713-0aaff964756f-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:55.200478 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:55.200240    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c30ec77-1b75-482e-a713-0aaff964756f-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:12:55.710738 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:55.710708    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c96595e9-95b1-4a92-a7ad-93932a2001ba" path="/var/lib/kubelet/pods/c96595e9-95b1-4a92-a7ad-93932a2001ba/volumes"
Oct 30 22:12:55.711094 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:55.711081    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcb28650-f1aa-4f60-a062-bd7dc84418eb" path="/var/lib/kubelet/pods/dcb28650-f1aa-4f60-a062-bd7dc84418eb/volumes"
Oct 30 22:12:55.820997 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:55.820966    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f84a2ad-bc4cc"
Oct 30 22:12:55.820997 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:55.820978    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f84a2ad-bc4cc" event={"ID":"6c30ec77-1b75-482e-a713-0aaff964756f","Type":"ContainerDied","Data":"1f48b3fa2c41e15bc92b7d38abd64b427fd9d09d4bbcdce57e09f8bc3d578b6b"}
Oct 30 22:12:55.821304 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:55.821038    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f48b3fa2c41e15bc92b7d38abd64b427fd9d09d4bbcdce57e09f8bc3d578b6b"
Oct 30 22:12:56.084104 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:56.084069    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"]
Oct 30 22:12:56.084543 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:12:56.084315    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429" podUID="86ab6a2a-10d7-4957-96ac-4cc29e0fb409" containerName="istio-proxy" containerID="cri-o://b2b6708421a1d098b8c12539dd710eb0c465c1ca04e8d912d91fb51c4715f4d0" gracePeriod=30
Oct 30 22:13:01.356411 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.356332    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 22:13:01.377081 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.377056    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istio-envoy\") pod \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") "
Oct 30 22:13:01.377192 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.377092    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istiod-ca-cert\") pod \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") "
Oct 30 22:13:01.377192 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.377139    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-credential-socket\") pod \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") "
Oct 30 22:13:01.377284 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.377222    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-kube-api-access-8czfh\" (UniqueName: \"kubernetes.io/projected/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-kube-api-access-8czfh\") pod \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") "
Oct 30 22:13:01.377329 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.377285    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istio-data\") pod \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") "
Oct 30 22:13:01.377407 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.377365    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istio-token\" (UniqueName: \"kubernetes.io/projected/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istio-token\") pod \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") "
Oct 30 22:13:01.377469 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.377430    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istio-podinfo\") pod \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") "
Oct 30 22:13:01.377526 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.377476    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-workload-certs\") pod \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") "
Oct 30 22:13:01.377526 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.377503    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-workload-socket\") pod \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\" (UID: \"86ab6a2a-10d7-4957-96ac-4cc29e0fb409\") "
Oct 30 22:13:01.377634 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.377572    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-credential-socket" pod "86ab6a2a-10d7-4957-96ac-4cc29e0fb409" (UID: "86ab6a2a-10d7-4957-96ac-4cc29e0fb409"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:01.377634 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.377577    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istiod-ca-cert" pod "86ab6a2a-10d7-4957-96ac-4cc29e0fb409" (UID: "86ab6a2a-10d7-4957-96ac-4cc29e0fb409"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:01.377843 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.377820    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-workload-certs" pod "86ab6a2a-10d7-4957-96ac-4cc29e0fb409" (UID: "86ab6a2a-10d7-4957-96ac-4cc29e0fb409"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:01.377935 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.377920    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istio-data" pod "86ab6a2a-10d7-4957-96ac-4cc29e0fb409" (UID: "86ab6a2a-10d7-4957-96ac-4cc29e0fb409"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:01.378014 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.377990    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-workload-socket" pod "86ab6a2a-10d7-4957-96ac-4cc29e0fb409" (UID: "86ab6a2a-10d7-4957-96ac-4cc29e0fb409"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:01.378401 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.378379    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:01.378495 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.378407    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:01.378495 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.378423    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:01.378495 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.378437    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:01.378495 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.378452    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:01.379486 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.379458    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istio-envoy" pod "86ab6a2a-10d7-4957-96ac-4cc29e0fb409" (UID: "86ab6a2a-10d7-4957-96ac-4cc29e0fb409"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:01.379727 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.379702    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istio-token" pod "86ab6a2a-10d7-4957-96ac-4cc29e0fb409" (UID: "86ab6a2a-10d7-4957-96ac-4cc29e0fb409"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:01.379822 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.379786    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istio-podinfo" pod "86ab6a2a-10d7-4957-96ac-4cc29e0fb409" (UID: "86ab6a2a-10d7-4957-96ac-4cc29e0fb409"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:13:01.379822 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.379809    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-kube-api-access-8czfh" pod "86ab6a2a-10d7-4957-96ac-4cc29e0fb409" (UID: "86ab6a2a-10d7-4957-96ac-4cc29e0fb409"). InnerVolumeSpecName "kube-api-access-8czfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:01.479476 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.479443    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:01.479476 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.479468    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8czfh\" (UniqueName: \"kubernetes.io/projected/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-kube-api-access-8czfh\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:01.479476 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.479479    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:01.479709 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.479488    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86ab6a2a-10d7-4957-96ac-4cc29e0fb409-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:01.864043 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.864005    5087 generic.go:356] "Generic (PLEG): container finished" podID="86ab6a2a-10d7-4957-96ac-4cc29e0fb409" containerID="b2b6708421a1d098b8c12539dd710eb0c465c1ca04e8d912d91fb51c4715f4d0" exitCode=0
Oct 30 22:13:01.864220 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.864066    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"
Oct 30 22:13:01.864220 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.864087    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429" event={"ID":"86ab6a2a-10d7-4957-96ac-4cc29e0fb409","Type":"ContainerDied","Data":"b2b6708421a1d098b8c12539dd710eb0c465c1ca04e8d912d91fb51c4715f4d0"}
Oct 30 22:13:01.864220 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.864147    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429" event={"ID":"86ab6a2a-10d7-4957-96ac-4cc29e0fb409","Type":"ContainerDied","Data":"86d8dc66fac68cbc9ee289dca70e6c85298a2fe872e5c139aadce50ee818c49f"}
Oct 30 22:13:01.864220 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.864174    5087 scope.go:122] "RemoveContainer" containerID="b2b6708421a1d098b8c12539dd710eb0c465c1ca04e8d912d91fb51c4715f4d0"
Oct 30 22:13:01.875165 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.875147    5087 scope.go:122] "RemoveContainer" containerID="b2b6708421a1d098b8c12539dd710eb0c465c1ca04e8d912d91fb51c4715f4d0"
Oct 30 22:13:01.875490 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:13:01.875469    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2b6708421a1d098b8c12539dd710eb0c465c1ca04e8d912d91fb51c4715f4d0\": container with ID starting with b2b6708421a1d098b8c12539dd710eb0c465c1ca04e8d912d91fb51c4715f4d0 not found: ID does not exist" containerID="b2b6708421a1d098b8c12539dd710eb0c465c1ca04e8d912d91fb51c4715f4d0"
Oct 30 22:13:01.875556 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:01.875497    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2b6708421a1d098b8c12539dd710eb0c465c1ca04e8d912d91fb51c4715f4d0"} err="failed to get container status \"b2b6708421a1d098b8c12539dd710eb0c465c1ca04e8d912d91fb51c4715f4d0\": rpc error: code = NotFound desc = could not find container \"b2b6708421a1d098b8c12539dd710eb0c465c1ca04e8d912d91fb51c4715f4d0\": container with ID starting with b2b6708421a1d098b8c12539dd710eb0c465c1ca04e8d912d91fb51c4715f4d0 not found: ID does not exist"
Oct 30 22:13:02.031470 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:02.031439    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"]
Oct 30 22:13:02.034337 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:02.034309    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-dda9f5-openshift-default-59dc47f975-rf429"]
Oct 30 22:13:03.708955 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:03.708913    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86ab6a2a-10d7-4957-96ac-4cc29e0fb409" path="/var/lib/kubelet/pods/86ab6a2a-10d7-4957-96ac-4cc29e0fb409/volumes"
Oct 30 22:13:03.839196 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:03.839168    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7051-c046bc"]
Oct 30 22:13:03.839792 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:03.839764    5087 cpu_manager.go:414] "containerMap: removing container" podUID="c96595e9-95b1-4a92-a7ad-93932a2001ba" containerName="payload-pre-processing"
Oct 30 22:13:03.839919 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:03.839794    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c96595e9-95b1-4a92-a7ad-93932a2001ba" containerName="payload-pre-processing"
Oct 30 22:13:03.839919 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:03.839832    5087 cpu_manager.go:414] "containerMap: removing container" podUID="6c30ec77-1b75-482e-a713-0aaff964756f" containerName="revoke-keys"
Oct 30 22:13:03.839919 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:03.839843    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c30ec77-1b75-482e-a713-0aaff964756f" containerName="revoke-keys"
Oct 30 22:13:03.839919 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:03.839865    5087 cpu_manager.go:414] "containerMap: removing container" podUID="86ab6a2a-10d7-4957-96ac-4cc29e0fb409" containerName="istio-proxy"
Oct 30 22:13:03.839919 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:03.839873    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86ab6a2a-10d7-4957-96ac-4cc29e0fb409" containerName="istio-proxy"
Oct 30 22:13:03.839919 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:03.839896    5087 cpu_manager.go:414] "containerMap: removing container" podUID="dcb28650-f1aa-4f60-a062-bd7dc84418eb" containerName="payload-processing"
Oct 30 22:13:03.839919 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:03.839904    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcb28650-f1aa-4f60-a062-bd7dc84418eb" containerName="payload-processing"
Oct 30 22:13:03.840151 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:03.840117    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c30ec77-1b75-482e-a713-0aaff964756f" containerName="revoke-keys"
Oct 30 22:13:03.840151 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:03.840133    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="86ab6a2a-10d7-4957-96ac-4cc29e0fb409" containerName="istio-proxy"
Oct 30 22:13:03.840216 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:03.840150    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcb28650-f1aa-4f60-a062-bd7dc84418eb" containerName="payload-processing"
Oct 30 22:13:03.840216 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:03.840169    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="c96595e9-95b1-4a92-a7ad-93932a2001ba" containerName="payload-pre-processing"
Oct 30 22:13:03.845040 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:03.845016    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7051-c046bc"
Oct 30 22:13:03.898411 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:03.898382    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7051-c046bc"]
Oct 30 22:13:03.909879 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:03.909851    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvnxz\" (UniqueName: \"kubernetes.io/projected/003e6f92-ae6e-49a4-8de2-692420d876f8-kube-api-access-zvnxz\") pod \"test-curl-7051-c046bc\" (UID: \"003e6f92-ae6e-49a4-8de2-692420d876f8\") " pod="opendatahub/test-curl-7051-c046bc"
Oct 30 22:13:03.944284 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:03.944247    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fs5vb\""
Oct 30 22:13:04.011685 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:04.011570    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvnxz\" (UniqueName: \"kubernetes.io/projected/003e6f92-ae6e-49a4-8de2-692420d876f8-kube-api-access-zvnxz\") pod \"test-curl-7051-c046bc\" (UID: \"003e6f92-ae6e-49a4-8de2-692420d876f8\") " pod="opendatahub/test-curl-7051-c046bc"
Oct 30 22:13:04.068867 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:04.068835    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvnxz\" (UniqueName: \"kubernetes.io/projected/003e6f92-ae6e-49a4-8de2-692420d876f8-kube-api-access-zvnxz\") pod \"test-curl-7051-c046bc\" (UID: \"003e6f92-ae6e-49a4-8de2-692420d876f8\") " pod="opendatahub/test-curl-7051-c046bc"
Oct 30 22:13:04.163529 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:04.163502    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7051-c046bc"
Oct 30 22:13:04.544418 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:04.544389    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7051-c046bc"]
Oct 30 22:13:04.545725 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:13:04.545701    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod003e6f92_ae6e_49a4_8de2_692420d876f8.slice/crio-9ef6b34d1ad90887a53fe05e3b3a637d2ffcdc39ab210fd98ab6e6fdfba36047 WatchSource:0}: Error finding container 9ef6b34d1ad90887a53fe05e3b3a637d2ffcdc39ab210fd98ab6e6fdfba36047: Status 404 returned error can't find the container with id 9ef6b34d1ad90887a53fe05e3b3a637d2ffcdc39ab210fd98ab6e6fdfba36047
Oct 30 22:13:04.890885 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:04.890779    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7051-c046bc" event={"ID":"003e6f92-ae6e-49a4-8de2-692420d876f8","Type":"ContainerStarted","Data":"72588716950b1b8b7ec4f4286d7911557d5358e7fb42f99242812fc6365b22a2"}
Oct 30 22:13:04.890885 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:04.890842    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7051-c046bc" event={"ID":"003e6f92-ae6e-49a4-8de2-692420d876f8","Type":"ContainerStarted","Data":"9ef6b34d1ad90887a53fe05e3b3a637d2ffcdc39ab210fd98ab6e6fdfba36047"}
Oct 30 22:13:05.006052 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:05.005998    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7051-c046bc" podStartSLOduration=2.005982692 podStartE2EDuration="2.005982692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:05.003908191 +0000 UTC m=+2379.478412171" watchObservedRunningTime="2026-10-30 22:13:05.005982692 +0000 UTC m=+2379.480486671"
Oct 30 22:13:06.496707 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:06.496664    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7051-c046bc"]
Oct 30 22:13:06.501893 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:06.501861    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7051-c046bc"]
Oct 30 22:13:06.904959 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:06.904836    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7051-c046bc" podUID="003e6f92-ae6e-49a4-8de2-692420d876f8" containerName="test-curl-7051-c046bc" containerID="cri-o://72588716950b1b8b7ec4f4286d7911557d5358e7fb42f99242812fc6365b22a2" gracePeriod=2
Oct 30 22:13:06.955709 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:13:06.955669    5087 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7051-c046bc\" is forbidden: User \"system:node:ip-10-0-132-92.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-92.us-west-1.compute.internal' and this object" podUID="003e6f92-ae6e-49a4-8de2-692420d876f8" pod="opendatahub/test-curl-7051-c046bc"
Oct 30 22:13:07.083190 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:07.083157    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-wqggm"]
Oct 30 22:13:07.083757 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:07.083742    5087 cpu_manager.go:414] "containerMap: removing container" podUID="003e6f92-ae6e-49a4-8de2-692420d876f8" containerName="test-curl-7051-c046bc"
Oct 30 22:13:07.083813 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:07.083758    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="003e6f92-ae6e-49a4-8de2-692420d876f8" containerName="test-curl-7051-c046bc"
Oct 30 22:13:07.083876 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:07.083867    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="003e6f92-ae6e-49a4-8de2-692420d876f8" containerName="test-curl-7051-c046bc"
Oct 30 22:13:07.087255 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:07.087238    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-wqggm"
Oct 30 22:13:07.138502 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:13:07.138462    5087 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7051-c046bc\" is forbidden: User \"system:node:ip-10-0-132-92.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-92.us-west-1.compute.internal' and this object" podUID="003e6f92-ae6e-49a4-8de2-692420d876f8" pod="opendatahub/test-curl-7051-c046bc"
Oct 30 22:13:07.145558 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:07.145504    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k2xh\" (UniqueName: \"kubernetes.io/projected/c57cf351-4009-4aaa-9ee6-c579ec319a7a-kube-api-access-4k2xh\") pod \"maas-api-revoke-keys-e2e-mt-6df3ba0e-wqggm\" (UID: \"c57cf351-4009-4aaa-9ee6-c579ec319a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-wqggm"
Oct 30 22:13:07.145686 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:07.145641    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c57cf351-4009-4aaa-9ee6-c579ec319a7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6df3ba0e-wqggm\" (UID: \"c57cf351-4009-4aaa-9ee6-c579ec319a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-wqggm"
Oct 30 22:13:07.145686 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:07.145675    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c57cf351-4009-4aaa-9ee6-c579ec319a7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6df3ba0e-wqggm\" (UID: \"c57cf351-4009-4aaa-9ee6-c579ec319a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-wqggm"
Oct 30 22:13:07.148029 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:07.148008    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-wqggm"]
Oct 30 22:13:07.188255 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:07.188164    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:13:07.193257 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:07.193207    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:13:07.246950 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:07.246912    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k2xh\" (UniqueName: \"kubernetes.io/projected/c57cf351-4009-4aaa-9ee6-c579ec319a7a-kube-api-access-4k2xh\") pod \"maas-api-revoke-keys-e2e-mt-6df3ba0e-wqggm\" (UID: \"c57cf351-4009-4aaa-9ee6-c579ec319a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-wqggm"
Oct 30 22:13:07.247090 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:07.247073    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c57cf351-4009-4aaa-9ee6-c579ec319a7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6df3ba0e-wqggm\" (UID: \"c57cf351-4009-4aaa-9ee6-c579ec319a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-wqggm"
Oct 30 22:13:07.247146 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:07.247117    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c57cf351-4009-4aaa-9ee6-c579ec319a7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6df3ba0e-wqggm\" (UID: \"c57cf351-4009-4aaa-9ee6-c579ec319a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-wqggm"
Oct 30 22:13:07.247829 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:07.247801    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c57cf351-4009-4aaa-9ee6-c579ec319a7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6df3ba0e-wqggm\" (UID: \"c57cf351-4009-4aaa-9ee6-c579ec319a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-wqggm"
Oct 30 22:13:07.249710 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:07.249692    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c57cf351-4009-4aaa-9ee6-c579ec319a7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6df3ba0e-wqggm\" (UID: \"c57cf351-4009-4aaa-9ee6-c579ec319a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-wqggm"
Oct 30 22:13:07.304989 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:07.304960    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k2xh\" (UniqueName: \"kubernetes.io/projected/c57cf351-4009-4aaa-9ee6-c579ec319a7a-kube-api-access-4k2xh\") pod \"maas-api-revoke-keys-e2e-mt-6df3ba0e-wqggm\" (UID: \"c57cf351-4009-4aaa-9ee6-c579ec319a7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-wqggm"
Oct 30 22:13:07.400641 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:07.400609    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-wqggm"
Oct 30 22:13:07.809151 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:13:07.809122    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57cf351_4009_4aaa_9ee6_c579ec319a7a.slice/crio-54f011f1f48106674789e49132b57c4154e72db8106f8a25eeb98f1ee323d64f WatchSource:0}: Error finding container 54f011f1f48106674789e49132b57c4154e72db8106f8a25eeb98f1ee323d64f: Status 404 returned error can't find the container with id 54f011f1f48106674789e49132b57c4154e72db8106f8a25eeb98f1ee323d64f
Oct 30 22:13:07.809504 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:07.809175    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-wqggm"]
Oct 30 22:13:07.913155 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:07.913125    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-wqggm" event={"ID":"c57cf351-4009-4aaa-9ee6-c579ec319a7a","Type":"ContainerStarted","Data":"54f011f1f48106674789e49132b57c4154e72db8106f8a25eeb98f1ee323d64f"}
Oct 30 22:13:08.921461 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:08.921431    5087 generic.go:356] "Generic (PLEG): container finished" podID="c57cf351-4009-4aaa-9ee6-c579ec319a7a" containerID="ef6aa7205da8bb246fcfe0dd9c02fde7813a71bcdea1351c9b227a2cc9929eda" exitCode=0
Oct 30 22:13:08.921808 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:08.921550    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-wqggm" event={"ID":"c57cf351-4009-4aaa-9ee6-c579ec319a7a","Type":"ContainerDied","Data":"ef6aa7205da8bb246fcfe0dd9c02fde7813a71bcdea1351c9b227a2cc9929eda"}
Oct 30 22:13:09.119703 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:09.119678    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7051-c046bc"
Oct 30 22:13:09.171797 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:09.171764    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/003e6f92-ae6e-49a4-8de2-692420d876f8-kube-api-access-zvnxz\" (UniqueName: \"kubernetes.io/projected/003e6f92-ae6e-49a4-8de2-692420d876f8-kube-api-access-zvnxz\") pod \"003e6f92-ae6e-49a4-8de2-692420d876f8\" (UID: \"003e6f92-ae6e-49a4-8de2-692420d876f8\") "
Oct 30 22:13:09.174117 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:09.174085    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/003e6f92-ae6e-49a4-8de2-692420d876f8-kube-api-access-zvnxz" pod "003e6f92-ae6e-49a4-8de2-692420d876f8" (UID: "003e6f92-ae6e-49a4-8de2-692420d876f8"). InnerVolumeSpecName "kube-api-access-zvnxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:09.273891 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:09.273855    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvnxz\" (UniqueName: \"kubernetes.io/projected/003e6f92-ae6e-49a4-8de2-692420d876f8-kube-api-access-zvnxz\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:09.709338 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:09.709305    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="003e6f92-ae6e-49a4-8de2-692420d876f8" path="/var/lib/kubelet/pods/003e6f92-ae6e-49a4-8de2-692420d876f8/volumes"
Oct 30 22:13:09.929912 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:09.929871    5087 generic.go:356] "Generic (PLEG): container finished" podID="003e6f92-ae6e-49a4-8de2-692420d876f8" containerID="72588716950b1b8b7ec4f4286d7911557d5358e7fb42f99242812fc6365b22a2" exitCode=137
Oct 30 22:13:09.929912 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:09.929915    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7051-c046bc"
Oct 30 22:13:09.930461 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:09.929966    5087 scope.go:122] "RemoveContainer" containerID="72588716950b1b8b7ec4f4286d7911557d5358e7fb42f99242812fc6365b22a2"
Oct 30 22:13:09.942066 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:09.942042    5087 scope.go:122] "RemoveContainer" containerID="72588716950b1b8b7ec4f4286d7911557d5358e7fb42f99242812fc6365b22a2"
Oct 30 22:13:09.942448 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:13:09.942425    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72588716950b1b8b7ec4f4286d7911557d5358e7fb42f99242812fc6365b22a2\": container with ID starting with 72588716950b1b8b7ec4f4286d7911557d5358e7fb42f99242812fc6365b22a2 not found: ID does not exist" containerID="72588716950b1b8b7ec4f4286d7911557d5358e7fb42f99242812fc6365b22a2"
Oct 30 22:13:09.942561 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:09.942460    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72588716950b1b8b7ec4f4286d7911557d5358e7fb42f99242812fc6365b22a2"} err="failed to get container status \"72588716950b1b8b7ec4f4286d7911557d5358e7fb42f99242812fc6365b22a2\": rpc error: code = NotFound desc = could not find container \"72588716950b1b8b7ec4f4286d7911557d5358e7fb42f99242812fc6365b22a2\": container with ID starting with 72588716950b1b8b7ec4f4286d7911557d5358e7fb42f99242812fc6365b22a2 not found: ID does not exist"
Oct 30 22:13:10.101193 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:10.101153    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-wqggm"
Oct 30 22:13:10.184252 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:10.184219    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c57cf351-4009-4aaa-9ee6-c579ec319a7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c57cf351-4009-4aaa-9ee6-c579ec319a7a-maas-api-openshift-ca\") pod \"c57cf351-4009-4aaa-9ee6-c579ec319a7a\" (UID: \"c57cf351-4009-4aaa-9ee6-c579ec319a7a\") "
Oct 30 22:13:10.184435 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:10.184401    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c57cf351-4009-4aaa-9ee6-c579ec319a7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c57cf351-4009-4aaa-9ee6-c579ec319a7a-maas-api-xks-ca\") pod \"c57cf351-4009-4aaa-9ee6-c579ec319a7a\" (UID: \"c57cf351-4009-4aaa-9ee6-c579ec319a7a\") "
Oct 30 22:13:10.184505 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:10.184484    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c57cf351-4009-4aaa-9ee6-c579ec319a7a-kube-api-access-4k2xh\" (UniqueName: \"kubernetes.io/projected/c57cf351-4009-4aaa-9ee6-c579ec319a7a-kube-api-access-4k2xh\") pod \"c57cf351-4009-4aaa-9ee6-c579ec319a7a\" (UID: \"c57cf351-4009-4aaa-9ee6-c579ec319a7a\") "
Oct 30 22:13:10.184615 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:10.184593    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c57cf351-4009-4aaa-9ee6-c579ec319a7a-maas-api-openshift-ca" pod "c57cf351-4009-4aaa-9ee6-c579ec319a7a" (UID: "c57cf351-4009-4aaa-9ee6-c579ec319a7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:10.184927 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:10.184907    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c57cf351-4009-4aaa-9ee6-c579ec319a7a-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:10.186755 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:10.186725    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c57cf351-4009-4aaa-9ee6-c579ec319a7a-maas-api-xks-ca" pod "c57cf351-4009-4aaa-9ee6-c579ec319a7a" (UID: "c57cf351-4009-4aaa-9ee6-c579ec319a7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:10.186867 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:10.186774    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c57cf351-4009-4aaa-9ee6-c579ec319a7a-kube-api-access-4k2xh" pod "c57cf351-4009-4aaa-9ee6-c579ec319a7a" (UID: "c57cf351-4009-4aaa-9ee6-c579ec319a7a"). InnerVolumeSpecName "kube-api-access-4k2xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:10.286110 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:10.286019    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c57cf351-4009-4aaa-9ee6-c579ec319a7a-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:10.286110 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:10.286055    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k2xh\" (UniqueName: \"kubernetes.io/projected/c57cf351-4009-4aaa-9ee6-c579ec319a7a-kube-api-access-4k2xh\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:10.891047 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:10.891011    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"]
Oct 30 22:13:10.891617 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:10.891598    5087 cpu_manager.go:414] "containerMap: removing container" podUID="c57cf351-4009-4aaa-9ee6-c579ec319a7a" containerName="revoke-keys"
Oct 30 22:13:10.891617 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:10.891616    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c57cf351-4009-4aaa-9ee6-c579ec319a7a" containerName="revoke-keys"
Oct 30 22:13:10.891817 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:10.891747    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="c57cf351-4009-4aaa-9ee6-c579ec319a7a" containerName="revoke-keys"
Oct 30 22:13:10.895529 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:10.895510    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:10.945467 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:10.945429    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-wqggm" event={"ID":"c57cf351-4009-4aaa-9ee6-c579ec319a7a","Type":"ContainerDied","Data":"54f011f1f48106674789e49132b57c4154e72db8106f8a25eeb98f1ee323d64f"}
Oct 30 22:13:10.945860 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:10.945487    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-wqggm"
Oct 30 22:13:10.945860 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:10.945489    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54f011f1f48106674789e49132b57c4154e72db8106f8a25eeb98f1ee323d64f"
Oct 30 22:13:10.947718 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:10.947693    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6df3ba0e-openshift-default-dockercfg-6ckx9\""
Oct 30 22:13:10.994159 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:10.994119    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-workload-certs\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:10.994159 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:10.994157    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5cg4\" (UniqueName: \"kubernetes.io/projected/53f51d9f-f90e-43f9-85de-eab0d05086ee-kube-api-access-h5cg4\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:10.994380 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:10.994288    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-istio-envoy\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:10.994469 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:10.994444    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53f51d9f-f90e-43f9-85de-eab0d05086ee-istiod-ca-cert\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:10.994543 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:10.994523    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53f51d9f-f90e-43f9-85de-eab0d05086ee-istio-token\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:10.994583 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:10.994564    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53f51d9f-f90e-43f9-85de-eab0d05086ee-istio-podinfo\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:10.994789 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:10.994686    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-credential-socket\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:10.994789 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:10.994718    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-istio-data\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:10.994789 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:10.994742    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-workload-socket\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:11.001456 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:11.001419    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"]
Oct 30 22:13:11.095679 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:11.095639    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-credential-socket\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:11.095880 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:11.095683    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-istio-data\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:11.095880 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:11.095711    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-workload-socket\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:11.095880 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:11.095776    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-workload-certs\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:11.095880 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:11.095799    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5cg4\" (UniqueName: \"kubernetes.io/projected/53f51d9f-f90e-43f9-85de-eab0d05086ee-kube-api-access-h5cg4\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:11.096110 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:11.095879    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-istio-envoy\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:11.096110 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:11.096089    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-credential-socket\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:11.096110 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:11.096095    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53f51d9f-f90e-43f9-85de-eab0d05086ee-istiod-ca-cert\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:11.096276 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:11.096128    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-istio-data\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:11.096276 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:11.096163    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-workload-socket\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:11.096276 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:11.096169    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53f51d9f-f90e-43f9-85de-eab0d05086ee-istio-token\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:11.096276 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:11.096236    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53f51d9f-f90e-43f9-85de-eab0d05086ee-istio-podinfo\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:11.096569 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:11.096274    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-workload-certs\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:11.096849 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:11.096826    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53f51d9f-f90e-43f9-85de-eab0d05086ee-istiod-ca-cert\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:11.098524 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:11.098500    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-istio-envoy\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:11.098785 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:11.098767    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53f51d9f-f90e-43f9-85de-eab0d05086ee-istio-podinfo\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:11.153112 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:11.153036    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53f51d9f-f90e-43f9-85de-eab0d05086ee-istio-token\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:11.153255 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:11.153200    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5cg4\" (UniqueName: \"kubernetes.io/projected/53f51d9f-f90e-43f9-85de-eab0d05086ee-kube-api-access-h5cg4\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:11.210666 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:11.210636    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:11.650287 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:11.650260    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"]
Oct 30 22:13:11.650764 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:13:11.650736    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f51d9f_f90e_43f9_85de_eab0d05086ee.slice/crio-6baa4ea4027c250b331b46e390f9318ff210da4cb181e5c9dc9d6c0238d7d7a7 WatchSource:0}: Error finding container 6baa4ea4027c250b331b46e390f9318ff210da4cb181e5c9dc9d6c0238d7d7a7: Status 404 returned error can't find the container with id 6baa4ea4027c250b331b46e390f9318ff210da4cb181e5c9dc9d6c0238d7d7a7
Oct 30 22:13:11.653566 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:11.653529    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:13:11.653667 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:11.653605    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:13:11.653667 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:11.653642    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:13:11.956269 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:11.956160    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54" event={"ID":"53f51d9f-f90e-43f9-85de-eab0d05086ee","Type":"ContainerStarted","Data":"c5bc22f224b29c496d2a94ba40586fa6be2a0e0e05eaa85da39ac6989a2c6e94"}
Oct 30 22:13:11.956269 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:11.956236    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54" event={"ID":"53f51d9f-f90e-43f9-85de-eab0d05086ee","Type":"ContainerStarted","Data":"6baa4ea4027c250b331b46e390f9318ff210da4cb181e5c9dc9d6c0238d7d7a7"}
Oct 30 22:13:12.211042 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:12.210956    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:12.212629 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:12.212607    5087 patch_prober.go:29] interesting pod/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 30 22:13:12.212696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:12.212654    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54" podUID="53f51d9f-f90e-43f9-85de-eab0d05086ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 30 22:13:13.211205 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:13.211168    5087 patch_prober.go:29] interesting pod/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 30 22:13:13.211637 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:13.211222    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54" podUID="53f51d9f-f90e-43f9-85de-eab0d05086ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 30 22:13:14.211643 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:14.211605    5087 patch_prober.go:29] interesting pod/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 30 22:13:14.212060 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:14.211659    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54" podUID="53f51d9f-f90e-43f9-85de-eab0d05086ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 30 22:13:15.211632 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:15.211595    5087 patch_prober.go:29] interesting pod/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 30 22:13:15.211833 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:15.211657    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54" podUID="53f51d9f-f90e-43f9-85de-eab0d05086ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 30 22:13:16.211943 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:16.211909    5087 patch_prober.go:29] interesting pod/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 30 22:13:16.212390 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:16.211964    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54" podUID="53f51d9f-f90e-43f9-85de-eab0d05086ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 30 22:13:17.192128 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:13:17.192088    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:13:17.210900 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:17.210871    5087 patch_prober.go:29] interesting pod/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 30 22:13:17.211083 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:17.210917    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54" podUID="53f51d9f-f90e-43f9-85de-eab0d05086ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 30 22:13:18.211144 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:18.211107    5087 patch_prober.go:29] interesting pod/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body=
Oct 30 22:13:18.211589 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:18.211160    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54" podUID="53f51d9f-f90e-43f9-85de-eab0d05086ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused"
Oct 30 22:13:18.813558 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:18.812625    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54" podStartSLOduration=8.812608192 podStartE2EDuration="8.812608192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:12.074760027 +0000 UTC m=+2386.549264010" watchObservedRunningTime="2026-10-30 22:13:18.812608192 +0000 UTC m=+2393.287112172"
Oct 30 22:13:18.815877 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:18.815856    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6df3ba0e-84d7646477-6qgzr"]
Oct 30 22:13:18.923569 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:18.923538    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6df3ba0e-7455bf6cc9-mcc6n"]
Oct 30 22:13:19.217704 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:19.217620    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:19.218438 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:19.218414    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:19.219249 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:19.219229    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:13:21.533373 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:21.531440    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded"
Oct 30 22:13:21.533373 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:21.532860    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:21.551384 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:21.546434    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:21.582394 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:21.576417    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml" podUID="01fdf382-6833-463a-bed9-c7c096166819" containerName="main" probeResult="failure" output="Get \"https://10.132.0.102:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:21.641372 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:21.638417    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:21.721946 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:21.721905    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main" probeResult="failure" output="Get \"https://10.132.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:23.816638 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:23.816565    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6df3ba0e-84d7646477-6qgzr" podUID="8834311d-0639-467b-b6a1-4dddd34f83de" containerName="payload-processing" containerID="cri-o://06cbaee83690f219920c6c7ca411592fd764b5b124dc4f91457f7a779336cce6" gracePeriod=25
Oct 30 22:13:23.934377 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:23.932546    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6df3ba0e-7455bf6cc9-mcc6n" podUID="a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a" containerName="payload-pre-processing" containerID="cri-o://eb91986c766b9fa5fb0d1c8b444d252471029e8581d5f481c35b0e9d8ac63470" gracePeriod=25
Oct 30 22:13:24.069953 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:24.069881    5087 generic.go:356] "Generic (PLEG): container finished" podID="a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a" containerID="eb91986c766b9fa5fb0d1c8b444d252471029e8581d5f481c35b0e9d8ac63470" exitCode=0
Oct 30 22:13:24.070096 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:24.069999    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6df3ba0e-7455bf6cc9-mcc6n" event={"ID":"a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a","Type":"ContainerDied","Data":"eb91986c766b9fa5fb0d1c8b444d252471029e8581d5f481c35b0e9d8ac63470"}
Oct 30 22:13:24.071765 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:24.071739    5087 generic.go:356] "Generic (PLEG): container finished" podID="8834311d-0639-467b-b6a1-4dddd34f83de" containerID="06cbaee83690f219920c6c7ca411592fd764b5b124dc4f91457f7a779336cce6" exitCode=0
Oct 30 22:13:24.072002 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:24.071911    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6df3ba0e-84d7646477-6qgzr" event={"ID":"8834311d-0639-467b-b6a1-4dddd34f83de","Type":"ContainerDied","Data":"06cbaee83690f219920c6c7ca411592fd764b5b124dc4f91457f7a779336cce6"}
Oct 30 22:13:24.144287 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:24.144259    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6df3ba0e-84d7646477-6qgzr"
Oct 30 22:13:24.239691 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:24.239665    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8834311d-0639-467b-b6a1-4dddd34f83de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8834311d-0639-467b-b6a1-4dddd34f83de-plugins-config-volume\") pod \"8834311d-0639-467b-b6a1-4dddd34f83de\" (UID: \"8834311d-0639-467b-b6a1-4dddd34f83de\") "
Oct 30 22:13:24.239832 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:24.239814    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8834311d-0639-467b-b6a1-4dddd34f83de-kube-api-access-66jd6\" (UniqueName: \"kubernetes.io/projected/8834311d-0639-467b-b6a1-4dddd34f83de-kube-api-access-66jd6\") pod \"8834311d-0639-467b-b6a1-4dddd34f83de\" (UID: \"8834311d-0639-467b-b6a1-4dddd34f83de\") "
Oct 30 22:13:24.240075 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:24.240049    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8834311d-0639-467b-b6a1-4dddd34f83de-plugins-config-volume" pod "8834311d-0639-467b-b6a1-4dddd34f83de" (UID: "8834311d-0639-467b-b6a1-4dddd34f83de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:24.240274 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:24.240251    5087 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8834311d-0639-467b-b6a1-4dddd34f83de-plugins-config-volume\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:24.241976 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:24.241951    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8834311d-0639-467b-b6a1-4dddd34f83de-kube-api-access-66jd6" pod "8834311d-0639-467b-b6a1-4dddd34f83de" (UID: "8834311d-0639-467b-b6a1-4dddd34f83de"). InnerVolumeSpecName "kube-api-access-66jd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:24.248525 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:24.248507    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6df3ba0e-7455bf6cc9-mcc6n"
Oct 30 22:13:24.341230 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:24.341154    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a-plugins-config-volume\") pod \"a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a\" (UID: \"a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a\") "
Oct 30 22:13:24.341416 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:24.341287    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a-kube-api-access-89gc7\" (UniqueName: \"kubernetes.io/projected/a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a-kube-api-access-89gc7\") pod \"a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a\" (UID: \"a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a\") "
Oct 30 22:13:24.341570 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:24.341547    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a-plugins-config-volume" pod "a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a" (UID: "a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:24.341702 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:24.341688    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66jd6\" (UniqueName: \"kubernetes.io/projected/8834311d-0639-467b-b6a1-4dddd34f83de-kube-api-access-66jd6\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:24.341748 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:24.341704    5087 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a-plugins-config-volume\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:24.343547 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:24.343524    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a-kube-api-access-89gc7" pod "a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a" (UID: "a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a"). InnerVolumeSpecName "kube-api-access-89gc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:24.443339 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:24.443308    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89gc7\" (UniqueName: \"kubernetes.io/projected/a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a-kube-api-access-89gc7\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:25.081193 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:25.081163    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6df3ba0e-7455bf6cc9-mcc6n"
Oct 30 22:13:25.081193 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:25.081177    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6df3ba0e-7455bf6cc9-mcc6n" event={"ID":"a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a","Type":"ContainerDied","Data":"108841a0bb45cedf2db7acef24c4964bbcb57faf80005ba177ce82732b6726eb"}
Oct 30 22:13:25.081773 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:25.081248    5087 scope.go:122] "RemoveContainer" containerID="eb91986c766b9fa5fb0d1c8b444d252471029e8581d5f481c35b0e9d8ac63470"
Oct 30 22:13:25.084303 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:25.084273    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6df3ba0e-84d7646477-6qgzr"
Oct 30 22:13:25.084456 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:25.084319    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6df3ba0e-84d7646477-6qgzr" event={"ID":"8834311d-0639-467b-b6a1-4dddd34f83de","Type":"ContainerDied","Data":"4bdca920021dc61988f374e03b95bed136a9664a7ed9580f8bca2b06c29cf2c3"}
Oct 30 22:13:25.093449 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:25.093408    5087 scope.go:122] "RemoveContainer" containerID="06cbaee83690f219920c6c7ca411592fd764b5b124dc4f91457f7a779336cce6"
Oct 30 22:13:25.245587 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:25.245557    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6df3ba0e-7455bf6cc9-mcc6n"]
Oct 30 22:13:25.295171 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:25.295141    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6df3ba0e-7455bf6cc9-mcc6n"]
Oct 30 22:13:25.362609 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:25.362536    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6df3ba0e-84d7646477-6qgzr"]
Oct 30 22:13:25.372513 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:25.372484    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6df3ba0e-84d7646477-6qgzr"]
Oct 30 22:13:25.709909 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:25.709837    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8834311d-0639-467b-b6a1-4dddd34f83de" path="/var/lib/kubelet/pods/8834311d-0639-467b-b6a1-4dddd34f83de/volumes"
Oct 30 22:13:25.710241 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:25.710228    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a" path="/var/lib/kubelet/pods/a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a/volumes"
Oct 30 22:13:26.016418 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.016380    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"]
Oct 30 22:13:26.016978 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.016956    5087 cpu_manager.go:414] "containerMap: removing container" podUID="a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a" containerName="payload-pre-processing"
Oct 30 22:13:26.017087 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.016979    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a" containerName="payload-pre-processing"
Oct 30 22:13:26.017087 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.017044    5087 cpu_manager.go:414] "containerMap: removing container" podUID="8834311d-0639-467b-b6a1-4dddd34f83de" containerName="payload-processing"
Oct 30 22:13:26.017087 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.017053    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8834311d-0639-467b-b6a1-4dddd34f83de" containerName="payload-processing"
Oct 30 22:13:26.017248 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.017198    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7218ba6-0a4c-4b1a-8698-9fbd18c95d2a" containerName="payload-pre-processing"
Oct 30 22:13:26.017248 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.017221    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="8834311d-0639-467b-b6a1-4dddd34f83de" containerName="payload-processing"
Oct 30 22:13:26.022175 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.022154    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:26.065868 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.065845    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"]
Oct 30 22:13:26.069554 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.069539    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:26.100266 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.100235    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"]
Oct 30 22:13:26.124399 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.124373    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 22:13:26.142699 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.142625    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 22:13:26.144994 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.144969    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 22:13:26.154460 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.154434    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 22:13:26.165639 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.165615    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db02748-f530-4034-b739-e9b4b9a263d5-istiod-ca-cert\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:26.165763 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.165646    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-istio-data\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:26.165763 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.165667    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-workload-socket\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:26.165763 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.165688    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf713fc3-b05d-453b-8e78-598e256b1e1b-istio-token\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:26.165872 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.165814    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-credential-socket\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:26.165872 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.165834    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jc56\" (UniqueName: \"kubernetes.io/projected/cf713fc3-b05d-453b-8e78-598e256b1e1b-kube-api-access-6jc56\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:26.165938 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.165923    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf713fc3-b05d-453b-8e78-598e256b1e1b-istio-podinfo\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:26.165973 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.165959    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxvtk\" (UniqueName: \"kubernetes.io/projected/5db02748-f530-4034-b739-e9b4b9a263d5-kube-api-access-wxvtk\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:26.166039 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.166026    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf713fc3-b05d-453b-8e78-598e256b1e1b-istiod-ca-cert\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:26.166089 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.166073    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db02748-f530-4034-b739-e9b4b9a263d5-istio-podinfo\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:26.166161 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.166140    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-workload-certs\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:26.166199 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.166173    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-istio-envoy\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:26.166252 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.166238    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-workload-socket\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:26.166288 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.166269    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-istio-data\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:26.166390 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.166375    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-credential-socket\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:26.166432 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.166412    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-workload-certs\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:26.166465 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.166432    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-istio-envoy\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:26.166465 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.166446    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db02748-f530-4034-b739-e9b4b9a263d5-istio-token\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:26.184511 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.184482    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"]
Oct 30 22:13:26.268256 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.268160    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-credential-socket\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:26.268256 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.268214    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-workload-certs\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:26.268256 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.268247    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-istio-envoy\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:26.268640 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.268272    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db02748-f530-4034-b739-e9b4b9a263d5-istio-token\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:26.268640 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.268316    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db02748-f530-4034-b739-e9b4b9a263d5-istiod-ca-cert\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:26.268640 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.268365    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-istio-data\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:26.268640 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.268399    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-workload-socket\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:26.268640 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.268438    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf713fc3-b05d-453b-8e78-598e256b1e1b-istio-token\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:26.268640 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.268508    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-credential-socket\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:26.268640 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.268535    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jc56\" (UniqueName: \"kubernetes.io/projected/cf713fc3-b05d-453b-8e78-598e256b1e1b-kube-api-access-6jc56\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:26.268640 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.268609    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf713fc3-b05d-453b-8e78-598e256b1e1b-istio-podinfo\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:26.268640 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.268630    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-credential-socket\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:26.269844 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.268676    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxvtk\" (UniqueName: \"kubernetes.io/projected/5db02748-f530-4034-b739-e9b4b9a263d5-kube-api-access-wxvtk\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:26.269844 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.268677    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-workload-certs\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:26.269844 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.268722    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf713fc3-b05d-453b-8e78-598e256b1e1b-istiod-ca-cert\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:26.269844 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.268792    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db02748-f530-4034-b739-e9b4b9a263d5-istio-podinfo\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:26.269844 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.268829    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-workload-certs\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:26.269844 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.268861    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-istio-envoy\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:26.269844 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.268918    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-workload-socket\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:26.269844 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.268961    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-istio-data\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:26.269844 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.268992    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-istio-data\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:26.269844 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.269058    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-workload-socket\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:26.269844 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.269328    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-workload-certs\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:26.269844 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.269374    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db02748-f530-4034-b739-e9b4b9a263d5-istiod-ca-cert\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:26.269844 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.269399    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-credential-socket\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:26.269844 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.269420    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-workload-socket\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:26.269844 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.269632    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-istio-data\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:26.270874 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.270497    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf713fc3-b05d-453b-8e78-598e256b1e1b-istiod-ca-cert\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:26.272687 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.272661    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-istio-envoy\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:26.272849 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.272829    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf713fc3-b05d-453b-8e78-598e256b1e1b-istio-podinfo\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:26.273046 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.273029    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-istio-envoy\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:26.273274 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.273255    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db02748-f530-4034-b739-e9b4b9a263d5-istio-podinfo\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:26.328872 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.328843    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf713fc3-b05d-453b-8e78-598e256b1e1b-istio-token\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:26.329050 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.328874    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxvtk\" (UniqueName: \"kubernetes.io/projected/5db02748-f530-4034-b739-e9b4b9a263d5-kube-api-access-wxvtk\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:26.329050 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.328906    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db02748-f530-4034-b739-e9b4b9a263d5-istio-token\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:26.329050 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.328911    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jc56\" (UniqueName: \"kubernetes.io/projected/cf713fc3-b05d-453b-8e78-598e256b1e1b-kube-api-access-6jc56\") pod \"e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") " pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:26.335848 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.335821    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:26.385121 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.384653    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:26.778015 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.777989    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"]
Oct 30 22:13:26.778508 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:13:26.778481    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db02748_f530_4034_b739_e9b4b9a263d5.slice/crio-655ae6ee888d31b458eb93ae21402f5c73a8c1fc124e1e527ca24f493b9eee35 WatchSource:0}: Error finding container 655ae6ee888d31b458eb93ae21402f5c73a8c1fc124e1e527ca24f493b9eee35: Status 404 returned error can't find the container with id 655ae6ee888d31b458eb93ae21402f5c73a8c1fc124e1e527ca24f493b9eee35
Oct 30 22:13:26.780954 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.780916    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:13:26.781049 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.780988    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:13:26.781049 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.781017    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:13:26.832981 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.832955    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"]
Oct 30 22:13:26.833281 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:13:26.833254    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf713fc3_b05d_453b_8e78_598e256b1e1b.slice/crio-1f4ec5e1df24e843ce1f10d7c53b67a5886641a6dac79900d6fa394ce420162c WatchSource:0}: Error finding container 1f4ec5e1df24e843ce1f10d7c53b67a5886641a6dac79900d6fa394ce420162c: Status 404 returned error can't find the container with id 1f4ec5e1df24e843ce1f10d7c53b67a5886641a6dac79900d6fa394ce420162c
Oct 30 22:13:26.836077 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.836042    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:13:26.836151 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.836110    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:13:26.836151 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:26.836138    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:13:27.104425 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:27.104081    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq" event={"ID":"5db02748-f530-4034-b739-e9b4b9a263d5","Type":"ContainerStarted","Data":"c85d8495b561e86ebe802aa367174df9e8391ac0a562ee115b1cb66439cdc08e"}
Oct 30 22:13:27.104425 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:27.104150    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq" event={"ID":"5db02748-f530-4034-b739-e9b4b9a263d5","Type":"ContainerStarted","Data":"655ae6ee888d31b458eb93ae21402f5c73a8c1fc124e1e527ca24f493b9eee35"}
Oct 30 22:13:27.107163 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:27.107134    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm" event={"ID":"cf713fc3-b05d-453b-8e78-598e256b1e1b","Type":"ContainerStarted","Data":"db81f7d4f8f112846732de3cd8924bb09227010a0e96dcce90761a10b9235256"}
Oct 30 22:13:27.107296 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:27.107188    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm" event={"ID":"cf713fc3-b05d-453b-8e78-598e256b1e1b","Type":"ContainerStarted","Data":"1f4ec5e1df24e843ce1f10d7c53b67a5886641a6dac79900d6fa394ce420162c"}
Oct 30 22:13:27.227434 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:27.223224    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq" podStartSLOduration=2.223204059 podStartE2EDuration="2.223204059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:27.222451044 +0000 UTC m=+2401.696955032" watchObservedRunningTime="2026-10-30 22:13:27.223204059 +0000 UTC m=+2401.697708038"
Oct 30 22:13:27.340371 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:27.339915    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:27.340371 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:27.340276    5087 patch_prober.go:29] interesting pod/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Oct 30 22:13:27.340371 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:27.340322    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq" podUID="5db02748-f530-4034-b739-e9b4b9a263d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Oct 30 22:13:27.351377 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:27.345107    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm" podStartSLOduration=2.345088639 podStartE2EDuration="2.345088639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:27.340690499 +0000 UTC m=+2401.815194544" watchObservedRunningTime="2026-10-30 22:13:27.345088639 +0000 UTC m=+2401.819592623"
Oct 30 22:13:27.386934 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:27.386841    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:27.388341 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:27.388307    5087 patch_prober.go:29] interesting pod/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Oct 30 22:13:27.388496 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:27.388376    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm" podUID="cf713fc3-b05d-453b-8e78-598e256b1e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Oct 30 22:13:28.347379 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:28.346737    5087 patch_prober.go:29] interesting pod/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Oct 30 22:13:28.347379 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:28.346794    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq" podUID="5db02748-f530-4034-b739-e9b4b9a263d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Oct 30 22:13:28.397374 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:28.392723    5087 patch_prober.go:29] interesting pod/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Oct 30 22:13:28.397374 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:28.392781    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm" podUID="cf713fc3-b05d-453b-8e78-598e256b1e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Oct 30 22:13:29.158373 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:29.143318    5087 scope.go:122] "RemoveContainer" containerID="2eb96bdd43dcbf0bf9cc88348babd0a34803f9262386e065df30df80637bbbdb"
Oct 30 22:13:29.357715 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:29.357680    5087 patch_prober.go:29] interesting pod/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Oct 30 22:13:29.369368 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:29.358220    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq" podUID="5db02748-f530-4034-b739-e9b4b9a263d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Oct 30 22:13:29.398372 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:29.395831    5087 patch_prober.go:29] interesting pod/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Oct 30 22:13:29.398372 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:29.395945    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm" podUID="cf713fc3-b05d-453b-8e78-598e256b1e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Oct 30 22:13:30.353374 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:30.352294    5087 patch_prober.go:29] interesting pod/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Oct 30 22:13:30.353374 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:30.352379    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq" podUID="5db02748-f530-4034-b739-e9b4b9a263d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Oct 30 22:13:30.399763 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:30.399726    5087 patch_prober.go:29] interesting pod/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Oct 30 22:13:30.400405 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:30.400363    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm" podUID="cf713fc3-b05d-453b-8e78-598e256b1e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Oct 30 22:13:30.584373 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:30.584085    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p"]
Oct 30 22:13:30.601370 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:30.599812    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-9b51e1-hqz6p"]
Oct 30 22:13:31.347727 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:31.347690    5087 patch_prober.go:29] interesting pod/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Oct 30 22:13:31.348009 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:31.347979    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq" podUID="5db02748-f530-4034-b739-e9b4b9a263d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Oct 30 22:13:31.385764 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:31.385730    5087 patch_prober.go:29] interesting pod/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Oct 30 22:13:31.386061 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:31.386030    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm" podUID="cf713fc3-b05d-453b-8e78-598e256b1e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Oct 30 22:13:31.712612 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:31.712528    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52b45001-b134-41d3-ae1c-c30ad4e265d4" path="/var/lib/kubelet/pods/52b45001-b134-41d3-ae1c-c30ad4e265d4/volumes"
Oct 30 22:13:32.336293 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:32.336258    5087 patch_prober.go:29] interesting pod/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused" start-of-body=
Oct 30 22:13:32.336605 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:32.336575    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq" podUID="5db02748-f530-4034-b739-e9b4b9a263d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.199:15021/healthz/ready\": dial tcp 10.132.0.199:15021: connect: connection refused"
Oct 30 22:13:32.388373 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:32.387740    5087 patch_prober.go:29] interesting pod/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Oct 30 22:13:32.388373 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:32.387808    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm" podUID="cf713fc3-b05d-453b-8e78-598e256b1e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Oct 30 22:13:33.385492 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:33.385449    5087 patch_prober.go:29] interesting pod/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused" start-of-body=
Oct 30 22:13:33.386010 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:33.385513    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm" podUID="cf713fc3-b05d-453b-8e78-598e256b1e1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.200:15021/healthz/ready\": dial tcp 10.132.0.200:15021: connect: connection refused"
Oct 30 22:13:33.441735 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:33.441699    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:33.442895 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:33.442867    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:33.446243 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:33.446216    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:13:34.390501 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:34.390469    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:34.390917 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:34.390874    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:34.391691 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:34.391670    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:13:34.633747 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:34.633714    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst"]
Oct 30 22:13:34.633953 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:34.633929    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst" podUID="c581dd8e-e662-4d4e-9ea8-21d19e46fcf0" containerName="maas-api" containerID="cri-o://211928ee6cce15b0182af607b9f0903a9a13bc6b9b7ca4693a0b856d671123d3" gracePeriod=30
Oct 30 22:13:34.992867 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:34.992835    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst"
Oct 30 22:13:35.189232 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:35.189193    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-metrics-certs\") pod \"c581dd8e-e662-4d4e-9ea8-21d19e46fcf0\" (UID: \"c581dd8e-e662-4d4e-9ea8-21d19e46fcf0\") "
Oct 30 22:13:35.189444 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:35.189240    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-openshift-service-ca\") pod \"c581dd8e-e662-4d4e-9ea8-21d19e46fcf0\" (UID: \"c581dd8e-e662-4d4e-9ea8-21d19e46fcf0\") "
Oct 30 22:13:35.189444 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:35.189393    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-kube-api-access-4vfxw\" (UniqueName: \"kubernetes.io/projected/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-kube-api-access-4vfxw\") pod \"c581dd8e-e662-4d4e-9ea8-21d19e46fcf0\" (UID: \"c581dd8e-e662-4d4e-9ea8-21d19e46fcf0\") "
Oct 30 22:13:35.189580 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:35.189456    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-maas-api-tls\") pod \"c581dd8e-e662-4d4e-9ea8-21d19e46fcf0\" (UID: \"c581dd8e-e662-4d4e-9ea8-21d19e46fcf0\") "
Oct 30 22:13:35.189663 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:35.189630    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-openshift-service-ca" pod "c581dd8e-e662-4d4e-9ea8-21d19e46fcf0" (UID: "c581dd8e-e662-4d4e-9ea8-21d19e46fcf0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:35.189947 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:35.189922    5087 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-openshift-service-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:35.191727 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:35.191698    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-kube-api-access-4vfxw" pod "c581dd8e-e662-4d4e-9ea8-21d19e46fcf0" (UID: "c581dd8e-e662-4d4e-9ea8-21d19e46fcf0"). InnerVolumeSpecName "kube-api-access-4vfxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:35.192133 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:35.192114    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-maas-api-tls" pod "c581dd8e-e662-4d4e-9ea8-21d19e46fcf0" (UID: "c581dd8e-e662-4d4e-9ea8-21d19e46fcf0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:35.192189 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:35.192145    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-metrics-certs" pod "c581dd8e-e662-4d4e-9ea8-21d19e46fcf0" (UID: "c581dd8e-e662-4d4e-9ea8-21d19e46fcf0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:35.241857 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:35.241820    5087 generic.go:356] "Generic (PLEG): container finished" podID="c581dd8e-e662-4d4e-9ea8-21d19e46fcf0" containerID="211928ee6cce15b0182af607b9f0903a9a13bc6b9b7ca4693a0b856d671123d3" exitCode=0
Oct 30 22:13:35.242121 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:35.241882    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst"
Oct 30 22:13:35.242121 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:35.241947    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst" event={"ID":"c581dd8e-e662-4d4e-9ea8-21d19e46fcf0","Type":"ContainerDied","Data":"211928ee6cce15b0182af607b9f0903a9a13bc6b9b7ca4693a0b856d671123d3"}
Oct 30 22:13:35.242121 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:35.241990    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst" event={"ID":"c581dd8e-e662-4d4e-9ea8-21d19e46fcf0","Type":"ContainerDied","Data":"98bd1e9165ed0f0b1ea8b488e17e03e7bf714bbe750a3295355f3687d2bc15b1"}
Oct 30 22:13:35.242121 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:35.242066    5087 scope.go:122] "RemoveContainer" containerID="211928ee6cce15b0182af607b9f0903a9a13bc6b9b7ca4693a0b856d671123d3"
Oct 30 22:13:35.264549 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:35.264418    5087 scope.go:122] "RemoveContainer" containerID="211928ee6cce15b0182af607b9f0903a9a13bc6b9b7ca4693a0b856d671123d3"
Oct 30 22:13:35.264884 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:13:35.264860    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"211928ee6cce15b0182af607b9f0903a9a13bc6b9b7ca4693a0b856d671123d3\": container with ID starting with 211928ee6cce15b0182af607b9f0903a9a13bc6b9b7ca4693a0b856d671123d3 not found: ID does not exist" containerID="211928ee6cce15b0182af607b9f0903a9a13bc6b9b7ca4693a0b856d671123d3"
Oct 30 22:13:35.265018 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:35.264899    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"211928ee6cce15b0182af607b9f0903a9a13bc6b9b7ca4693a0b856d671123d3"} err="failed to get container status \"211928ee6cce15b0182af607b9f0903a9a13bc6b9b7ca4693a0b856d671123d3\": rpc error: code = NotFound desc = could not find container \"211928ee6cce15b0182af607b9f0903a9a13bc6b9b7ca4693a0b856d671123d3\": container with ID starting with 211928ee6cce15b0182af607b9f0903a9a13bc6b9b7ca4693a0b856d671123d3 not found: ID does not exist"
Oct 30 22:13:35.291152 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:35.291080    5087 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-metrics-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:35.291322 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:35.291154    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vfxw\" (UniqueName: \"kubernetes.io/projected/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-kube-api-access-4vfxw\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:35.291322 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:35.291191    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0-maas-api-tls\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:35.406541 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:35.406507    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst"]
Oct 30 22:13:35.410675 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:35.410651    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-307368-6cbd4fd6cc-67sst"]
Oct 30 22:13:35.713213 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:35.713115    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c581dd8e-e662-4d4e-9ea8-21d19e46fcf0" path="/var/lib/kubelet/pods/c581dd8e-e662-4d4e-9ea8-21d19e46fcf0/volumes"
Oct 30 22:13:36.744422 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:36.744390    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-307368-xcfts"]
Oct 30 22:13:36.793685 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:36.793653    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-307368-xcfts"]
Oct 30 22:13:37.710025 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:37.709979    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b82f951-797b-437e-94c8-4892364f426f" path="/var/lib/kubelet/pods/6b82f951-797b-437e-94c8-4892364f426f/volumes"
Oct 30 22:13:38.602919 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:38.602889    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"]
Oct 30 22:13:38.603375 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:38.603103    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc" podUID="f95e4084-7a0f-4fa3-ad48-e34ee3eab117" containerName="istio-proxy" containerID="cri-o://520272e544a5110764fcfce3dbc31ba6560ecb5be68a621470878be98d087f74" gracePeriod=30
Oct 30 22:13:41.568537 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:41.568499    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb"]
Oct 30 22:13:41.569118 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:41.569101    5087 cpu_manager.go:414] "containerMap: removing container" podUID="c581dd8e-e662-4d4e-9ea8-21d19e46fcf0" containerName="maas-api"
Oct 30 22:13:41.569118 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:41.569118    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c581dd8e-e662-4d4e-9ea8-21d19e46fcf0" containerName="maas-api"
Oct 30 22:13:41.569268 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:41.569237    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="c581dd8e-e662-4d4e-9ea8-21d19e46fcf0" containerName="maas-api"
Oct 30 22:13:41.574185 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:41.574165    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb"
Oct 30 22:13:41.628754 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:41.628718    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-954628-serving-cert\""
Oct 30 22:13:41.681680 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:41.681648    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb"]
Oct 30 22:13:41.688983 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:41.688957    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe364587-c942-40b4-ae87-10c7cc1176b4-metrics-certs\") pod \"maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb\" (UID: \"fe364587-c942-40b4-ae87-10c7cc1176b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb"
Oct 30 22:13:41.689116 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:41.689025    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9nzm\" (UniqueName: \"kubernetes.io/projected/fe364587-c942-40b4-ae87-10c7cc1176b4-kube-api-access-h9nzm\") pod \"maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb\" (UID: \"fe364587-c942-40b4-ae87-10c7cc1176b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb"
Oct 30 22:13:41.689239 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:41.689224    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe364587-c942-40b4-ae87-10c7cc1176b4-maas-api-tls\") pod \"maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb\" (UID: \"fe364587-c942-40b4-ae87-10c7cc1176b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb"
Oct 30 22:13:41.689307 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:41.689256    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe364587-c942-40b4-ae87-10c7cc1176b4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb\" (UID: \"fe364587-c942-40b4-ae87-10c7cc1176b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb"
Oct 30 22:13:41.791020 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:41.790967    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe364587-c942-40b4-ae87-10c7cc1176b4-maas-api-tls\") pod \"maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb\" (UID: \"fe364587-c942-40b4-ae87-10c7cc1176b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb"
Oct 30 22:13:41.791207 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:41.791034    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe364587-c942-40b4-ae87-10c7cc1176b4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb\" (UID: \"fe364587-c942-40b4-ae87-10c7cc1176b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb"
Oct 30 22:13:41.791207 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:41.791099    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe364587-c942-40b4-ae87-10c7cc1176b4-metrics-certs\") pod \"maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb\" (UID: \"fe364587-c942-40b4-ae87-10c7cc1176b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb"
Oct 30 22:13:41.791207 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:13:41.791121    5087 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-serving-cert: secret "maas-api-e2e-inf-b-954628-serving-cert" not found
Oct 30 22:13:41.791207 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:41.791158    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9nzm\" (UniqueName: \"kubernetes.io/projected/fe364587-c942-40b4-ae87-10c7cc1176b4-kube-api-access-h9nzm\") pod \"maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb\" (UID: \"fe364587-c942-40b4-ae87-10c7cc1176b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb"
Oct 30 22:13:41.791207 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:13:41.791201    5087 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe364587-c942-40b4-ae87-10c7cc1176b4-maas-api-tls podName:fe364587-c942-40b4-ae87-10c7cc1176b4 nodeName:}" failed. No retries permitted until 2026-10-30 22:13:42.291178766 +0000 UTC m=+2416.765682740 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fe364587-c942-40b4-ae87-10c7cc1176b4-maas-api-tls") pod "maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb" (UID: "fe364587-c942-40b4-ae87-10c7cc1176b4") : secret "maas-api-e2e-inf-b-954628-serving-cert" not found
Oct 30 22:13:41.791853 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:41.791828    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe364587-c942-40b4-ae87-10c7cc1176b4-openshift-service-ca\") pod \"maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb\" (UID: \"fe364587-c942-40b4-ae87-10c7cc1176b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb"
Oct 30 22:13:41.793874 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:41.793855    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe364587-c942-40b4-ae87-10c7cc1176b4-metrics-certs\") pod \"maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb\" (UID: \"fe364587-c942-40b4-ae87-10c7cc1176b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb"
Oct 30 22:13:41.852825 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:41.852751    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9nzm\" (UniqueName: \"kubernetes.io/projected/fe364587-c942-40b4-ae87-10c7cc1176b4-kube-api-access-h9nzm\") pod \"maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb\" (UID: \"fe364587-c942-40b4-ae87-10c7cc1176b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb"
Oct 30 22:13:42.298071 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:42.298034    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe364587-c942-40b4-ae87-10c7cc1176b4-maas-api-tls\") pod \"maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb\" (UID: \"fe364587-c942-40b4-ae87-10c7cc1176b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb"
Oct 30 22:13:42.300653 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:42.300624    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe364587-c942-40b4-ae87-10c7cc1176b4-maas-api-tls\") pod \"maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb\" (UID: \"fe364587-c942-40b4-ae87-10c7cc1176b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb"
Oct 30 22:13:42.489801 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:42.489765    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb"
Oct 30 22:13:42.881498 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:42.881469    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb"]
Oct 30 22:13:42.882086 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:13:42.882058    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe364587_c942_40b4_ae87_10c7cc1176b4.slice/crio-1f48943c3119aec9838e6ccdea48e74a0a1b5adf6f370d1741a8d0c20c71ab2b WatchSource:0}: Error finding container 1f48943c3119aec9838e6ccdea48e74a0a1b5adf6f370d1741a8d0c20c71ab2b: Status 404 returned error can't find the container with id 1f48943c3119aec9838e6ccdea48e74a0a1b5adf6f370d1741a8d0c20c71ab2b
Oct 30 22:13:42.883430 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:42.883413    5087 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:13:43.301373 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:43.301307    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb" event={"ID":"fe364587-c942-40b4-ae87-10c7cc1176b4","Type":"ContainerStarted","Data":"1f48943c3119aec9838e6ccdea48e74a0a1b5adf6f370d1741a8d0c20c71ab2b"}
Oct 30 22:13:43.918887 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:43.918855    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 22:13:44.018814 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.018765    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istio-envoy\") pod \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") "
Oct 30 22:13:44.019029 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.018824    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istio-data\") pod \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") "
Oct 30 22:13:44.019029 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.018926    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-kube-api-access-w8w7c\" (UniqueName: \"kubernetes.io/projected/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-kube-api-access-w8w7c\") pod \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") "
Oct 30 22:13:44.019160 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.019057    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-credential-socket\") pod \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") "
Oct 30 22:13:44.019160 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.019062    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istio-data" pod "f95e4084-7a0f-4fa3-ad48-e34ee3eab117" (UID: "f95e4084-7a0f-4fa3-ad48-e34ee3eab117"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:44.019160 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.019093    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istio-token\" (UniqueName: \"kubernetes.io/projected/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istio-token\") pod \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") "
Oct 30 22:13:44.019160 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.019136    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-workload-certs\") pod \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") "
Oct 30 22:13:44.019429 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.019204    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istio-podinfo\") pod \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") "
Oct 30 22:13:44.019429 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.019267    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-workload-socket\") pod \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") "
Oct 30 22:13:44.019429 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.019294    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istiod-ca-cert\") pod \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\" (UID: \"f95e4084-7a0f-4fa3-ad48-e34ee3eab117\") "
Oct 30 22:13:44.020296 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.019847    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:44.020296 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.019941    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-workload-socket" pod "f95e4084-7a0f-4fa3-ad48-e34ee3eab117" (UID: "f95e4084-7a0f-4fa3-ad48-e34ee3eab117"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:44.020296 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.020057    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-workload-certs" pod "f95e4084-7a0f-4fa3-ad48-e34ee3eab117" (UID: "f95e4084-7a0f-4fa3-ad48-e34ee3eab117"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:44.020296 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.020177    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istiod-ca-cert" pod "f95e4084-7a0f-4fa3-ad48-e34ee3eab117" (UID: "f95e4084-7a0f-4fa3-ad48-e34ee3eab117"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:44.020623 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.020323    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-credential-socket" pod "f95e4084-7a0f-4fa3-ad48-e34ee3eab117" (UID: "f95e4084-7a0f-4fa3-ad48-e34ee3eab117"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:44.021843 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.021820    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istio-envoy" pod "f95e4084-7a0f-4fa3-ad48-e34ee3eab117" (UID: "f95e4084-7a0f-4fa3-ad48-e34ee3eab117"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:44.021843 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.021844    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istio-token" pod "f95e4084-7a0f-4fa3-ad48-e34ee3eab117" (UID: "f95e4084-7a0f-4fa3-ad48-e34ee3eab117"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:44.022037 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.021997    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istio-podinfo" pod "f95e4084-7a0f-4fa3-ad48-e34ee3eab117" (UID: "f95e4084-7a0f-4fa3-ad48-e34ee3eab117"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:13:44.022142 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.022123    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-kube-api-access-w8w7c" pod "f95e4084-7a0f-4fa3-ad48-e34ee3eab117" (UID: "f95e4084-7a0f-4fa3-ad48-e34ee3eab117"). InnerVolumeSpecName "kube-api-access-w8w7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:44.121620 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.121067    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8w7c\" (UniqueName: \"kubernetes.io/projected/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-kube-api-access-w8w7c\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:44.121620 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.121107    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:44.121620 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.121122    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:44.121620 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.121136    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:44.121620 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.121151    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:44.121620 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.121165    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:44.121620 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.121178    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:44.121620 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.121192    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f95e4084-7a0f-4fa3-ad48-e34ee3eab117-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:44.309523 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.309488    5087 generic.go:356] "Generic (PLEG): container finished" podID="f95e4084-7a0f-4fa3-ad48-e34ee3eab117" containerID="520272e544a5110764fcfce3dbc31ba6560ecb5be68a621470878be98d087f74" exitCode=0
Oct 30 22:13:44.309683 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.309531    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"
Oct 30 22:13:44.309683 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.309574    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc" event={"ID":"f95e4084-7a0f-4fa3-ad48-e34ee3eab117","Type":"ContainerDied","Data":"520272e544a5110764fcfce3dbc31ba6560ecb5be68a621470878be98d087f74"}
Oct 30 22:13:44.309683 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.309627    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc" event={"ID":"f95e4084-7a0f-4fa3-ad48-e34ee3eab117","Type":"ContainerDied","Data":"20d196bdf030d33d8d46abdce089cf495e14018e890238cf9003fe34ca2b2896"}
Oct 30 22:13:44.309683 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.309653    5087 scope.go:122] "RemoveContainer" containerID="520272e544a5110764fcfce3dbc31ba6560ecb5be68a621470878be98d087f74"
Oct 30 22:13:44.311526 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.311502    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb" event={"ID":"fe364587-c942-40b4-ae87-10c7cc1176b4","Type":"ContainerStarted","Data":"3c8dd812c0543cd619e975e8589844bf0ccfb918466ef5f47e7a942b6cdbaa4e"}
Oct 30 22:13:44.311663 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.311629    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb"
Oct 30 22:13:44.320888 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.320871    5087 scope.go:122] "RemoveContainer" containerID="520272e544a5110764fcfce3dbc31ba6560ecb5be68a621470878be98d087f74"
Oct 30 22:13:44.321140 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:13:44.321125    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"520272e544a5110764fcfce3dbc31ba6560ecb5be68a621470878be98d087f74\": container with ID starting with 520272e544a5110764fcfce3dbc31ba6560ecb5be68a621470878be98d087f74 not found: ID does not exist" containerID="520272e544a5110764fcfce3dbc31ba6560ecb5be68a621470878be98d087f74"
Oct 30 22:13:44.321182 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.321147    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"520272e544a5110764fcfce3dbc31ba6560ecb5be68a621470878be98d087f74"} err="failed to get container status \"520272e544a5110764fcfce3dbc31ba6560ecb5be68a621470878be98d087f74\": rpc error: code = NotFound desc = could not find container \"520272e544a5110764fcfce3dbc31ba6560ecb5be68a621470878be98d087f74\": container with ID starting with 520272e544a5110764fcfce3dbc31ba6560ecb5be68a621470878be98d087f74 not found: ID does not exist"
Oct 30 22:13:44.473958 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.473920    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"]
Oct 30 22:13:44.523876 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.523848    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9b51e1-openshift-default-6c7fc667bf-x86qc"]
Oct 30 22:13:44.573213 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.573123    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"]
Oct 30 22:13:44.573520 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.573487    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz" podUID="164e63e1-b309-4236-983a-2fd171af5b14" containerName="istio-proxy" containerID="cri-o://71bff2f05db10d65035ba080c86c05ff25df7ad6d47129088ed2b63d400c7f6d" gracePeriod=30
Oct 30 22:13:44.598023 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:44.597956    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb" podStartSLOduration=2.735224666 podStartE2EDuration="3.597936167s" totalImagesPullingTime="862.711501ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:44.591144698 +0000 UTC m=+2419.065648678" watchObservedRunningTime="2026-10-30 22:13:44.597936167 +0000 UTC m=+2419.072440154"
Oct 30 22:13:45.652567 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:45.652530    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b"]
Oct 30 22:13:45.653085 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:45.653070    5087 cpu_manager.go:414] "containerMap: removing container" podUID="f95e4084-7a0f-4fa3-ad48-e34ee3eab117" containerName="istio-proxy"
Oct 30 22:13:45.653085 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:45.653084    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f95e4084-7a0f-4fa3-ad48-e34ee3eab117" containerName="istio-proxy"
Oct 30 22:13:45.653207 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:45.653202    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="f95e4084-7a0f-4fa3-ad48-e34ee3eab117" containerName="istio-proxy"
Oct 30 22:13:45.658099 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:45.658075    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b"
Oct 30 22:13:45.710038 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:45.710000    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f95e4084-7a0f-4fa3-ad48-e34ee3eab117" path="/var/lib/kubelet/pods/f95e4084-7a0f-4fa3-ad48-e34ee3eab117/volumes"
Oct 30 22:13:45.713057 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:45.713030    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b"]
Oct 30 22:13:45.743305 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:45.743274    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fe89e6c-771c-4cb7-a4f5-54668eee2524-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b\" (UID: \"0fe89e6c-771c-4cb7-a4f5-54668eee2524\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b"
Oct 30 22:13:45.743554 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:45.743403    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fe89e6c-771c-4cb7-a4f5-54668eee2524-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b\" (UID: \"0fe89e6c-771c-4cb7-a4f5-54668eee2524\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b"
Oct 30 22:13:45.743554 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:45.743544    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8mhh\" (UniqueName: \"kubernetes.io/projected/0fe89e6c-771c-4cb7-a4f5-54668eee2524-kube-api-access-v8mhh\") pod \"maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b\" (UID: \"0fe89e6c-771c-4cb7-a4f5-54668eee2524\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b"
Oct 30 22:13:45.757281 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:45.757243    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:13:45.757429 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:45.757290    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:13:45.844536 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:45.844505    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8mhh\" (UniqueName: \"kubernetes.io/projected/0fe89e6c-771c-4cb7-a4f5-54668eee2524-kube-api-access-v8mhh\") pod \"maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b\" (UID: \"0fe89e6c-771c-4cb7-a4f5-54668eee2524\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b"
Oct 30 22:13:45.844709 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:45.844612    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fe89e6c-771c-4cb7-a4f5-54668eee2524-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b\" (UID: \"0fe89e6c-771c-4cb7-a4f5-54668eee2524\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b"
Oct 30 22:13:45.844709 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:45.844677    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fe89e6c-771c-4cb7-a4f5-54668eee2524-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b\" (UID: \"0fe89e6c-771c-4cb7-a4f5-54668eee2524\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b"
Oct 30 22:13:45.845254 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:45.845234    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fe89e6c-771c-4cb7-a4f5-54668eee2524-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b\" (UID: \"0fe89e6c-771c-4cb7-a4f5-54668eee2524\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b"
Oct 30 22:13:45.847190 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:45.847163    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fe89e6c-771c-4cb7-a4f5-54668eee2524-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b\" (UID: \"0fe89e6c-771c-4cb7-a4f5-54668eee2524\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b"
Oct 30 22:13:45.901133 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:45.901103    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8mhh\" (UniqueName: \"kubernetes.io/projected/0fe89e6c-771c-4cb7-a4f5-54668eee2524-kube-api-access-v8mhh\") pod \"maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b\" (UID: \"0fe89e6c-771c-4cb7-a4f5-54668eee2524\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b"
Oct 30 22:13:45.972509 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:45.972429    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b"
Oct 30 22:13:46.349179 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:46.349152    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b"]
Oct 30 22:13:46.350757 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:13:46.350735    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fe89e6c_771c_4cb7_a4f5_54668eee2524.slice/crio-52496c080b564a2108f16639d94ee6cec87d32355ff7f21196fca6d26abd88d3 WatchSource:0}: Error finding container 52496c080b564a2108f16639d94ee6cec87d32355ff7f21196fca6d26abd88d3: Status 404 returned error can't find the container with id 52496c080b564a2108f16639d94ee6cec87d32355ff7f21196fca6d26abd88d3
Oct 30 22:13:47.168209 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:13:47.168177    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:13:47.339278 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:47.339243    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b" event={"ID":"0fe89e6c-771c-4cb7-a4f5-54668eee2524","Type":"ContainerStarted","Data":"819e9b07f60279c3bf7772e1a561651b93b0d9e739c3e5ac33d93315895af2e0"}
Oct 30 22:13:47.339464 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:47.339299    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b" event={"ID":"0fe89e6c-771c-4cb7-a4f5-54668eee2524","Type":"ContainerStarted","Data":"52496c080b564a2108f16639d94ee6cec87d32355ff7f21196fca6d26abd88d3"}
Oct 30 22:13:47.471064 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:47.470965    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b" podStartSLOduration=2.470948005 podStartE2EDuration="2.470948005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:47.470131257 +0000 UTC m=+2421.944635236" watchObservedRunningTime="2026-10-30 22:13:47.470948005 +0000 UTC m=+2421.945451985"
Oct 30 22:13:48.347775 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:48.347741    5087 generic.go:356] "Generic (PLEG): container finished" podID="0fe89e6c-771c-4cb7-a4f5-54668eee2524" containerID="819e9b07f60279c3bf7772e1a561651b93b0d9e739c3e5ac33d93315895af2e0" exitCode=0
Oct 30 22:13:48.348177 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:48.347825    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b" event={"ID":"0fe89e6c-771c-4cb7-a4f5-54668eee2524","Type":"ContainerDied","Data":"819e9b07f60279c3bf7772e1a561651b93b0d9e739c3e5ac33d93315895af2e0"}
Oct 30 22:13:49.524840 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.524814    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b"
Oct 30 22:13:49.590246 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.590211    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fe89e6c-771c-4cb7-a4f5-54668eee2524-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fe89e6c-771c-4cb7-a4f5-54668eee2524-maas-api-xks-ca\") pod \"0fe89e6c-771c-4cb7-a4f5-54668eee2524\" (UID: \"0fe89e6c-771c-4cb7-a4f5-54668eee2524\") "
Oct 30 22:13:49.590449 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.590329    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fe89e6c-771c-4cb7-a4f5-54668eee2524-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fe89e6c-771c-4cb7-a4f5-54668eee2524-maas-api-openshift-ca\") pod \"0fe89e6c-771c-4cb7-a4f5-54668eee2524\" (UID: \"0fe89e6c-771c-4cb7-a4f5-54668eee2524\") "
Oct 30 22:13:49.590449 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.590429    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fe89e6c-771c-4cb7-a4f5-54668eee2524-kube-api-access-v8mhh\" (UniqueName: \"kubernetes.io/projected/0fe89e6c-771c-4cb7-a4f5-54668eee2524-kube-api-access-v8mhh\") pod \"0fe89e6c-771c-4cb7-a4f5-54668eee2524\" (UID: \"0fe89e6c-771c-4cb7-a4f5-54668eee2524\") "
Oct 30 22:13:49.590686 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.590651    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fe89e6c-771c-4cb7-a4f5-54668eee2524-maas-api-openshift-ca" pod "0fe89e6c-771c-4cb7-a4f5-54668eee2524" (UID: "0fe89e6c-771c-4cb7-a4f5-54668eee2524"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:49.590946 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.590917    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fe89e6c-771c-4cb7-a4f5-54668eee2524-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:49.592541 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.592518    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fe89e6c-771c-4cb7-a4f5-54668eee2524-maas-api-xks-ca" pod "0fe89e6c-771c-4cb7-a4f5-54668eee2524" (UID: "0fe89e6c-771c-4cb7-a4f5-54668eee2524"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:49.593035 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.593020    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fe89e6c-771c-4cb7-a4f5-54668eee2524-kube-api-access-v8mhh" pod "0fe89e6c-771c-4cb7-a4f5-54668eee2524" (UID: "0fe89e6c-771c-4cb7-a4f5-54668eee2524"). InnerVolumeSpecName "kube-api-access-v8mhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:49.692104 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.692066    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fe89e6c-771c-4cb7-a4f5-54668eee2524-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:49.692104 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.692107    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8mhh\" (UniqueName: \"kubernetes.io/projected/0fe89e6c-771c-4cb7-a4f5-54668eee2524-kube-api-access-v8mhh\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:49.844789 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.844759    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:13:49.995670 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.995628    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-workload-socket\") pod \"164e63e1-b309-4236-983a-2fd171af5b14\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") "
Oct 30 22:13:49.995870 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.995722    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/164e63e1-b309-4236-983a-2fd171af5b14-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/164e63e1-b309-4236-983a-2fd171af5b14-istio-podinfo\") pod \"164e63e1-b309-4236-983a-2fd171af5b14\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") "
Oct 30 22:13:49.995870 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.995761    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-workload-certs\") pod \"164e63e1-b309-4236-983a-2fd171af5b14\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") "
Oct 30 22:13:49.995870 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.995843    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-istio-envoy\") pod \"164e63e1-b309-4236-983a-2fd171af5b14\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") "
Oct 30 22:13:49.996071 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.995899    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-workload-socket" pod "164e63e1-b309-4236-983a-2fd171af5b14" (UID: "164e63e1-b309-4236-983a-2fd171af5b14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:49.996071 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.995926    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/164e63e1-b309-4236-983a-2fd171af5b14-kube-api-access-v9l5v\" (UniqueName: \"kubernetes.io/projected/164e63e1-b309-4236-983a-2fd171af5b14-kube-api-access-v9l5v\") pod \"164e63e1-b309-4236-983a-2fd171af5b14\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") "
Oct 30 22:13:49.996071 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.995966    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-istio-data\") pod \"164e63e1-b309-4236-983a-2fd171af5b14\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") "
Oct 30 22:13:49.996071 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.996020    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/164e63e1-b309-4236-983a-2fd171af5b14-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/164e63e1-b309-4236-983a-2fd171af5b14-istiod-ca-cert\") pod \"164e63e1-b309-4236-983a-2fd171af5b14\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") "
Oct 30 22:13:49.996294 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.996081    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-credential-socket\") pod \"164e63e1-b309-4236-983a-2fd171af5b14\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") "
Oct 30 22:13:49.996294 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.996102    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-workload-certs" pod "164e63e1-b309-4236-983a-2fd171af5b14" (UID: "164e63e1-b309-4236-983a-2fd171af5b14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:49.996294 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.996148    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/164e63e1-b309-4236-983a-2fd171af5b14-istio-token\" (UniqueName: \"kubernetes.io/projected/164e63e1-b309-4236-983a-2fd171af5b14-istio-token\") pod \"164e63e1-b309-4236-983a-2fd171af5b14\" (UID: \"164e63e1-b309-4236-983a-2fd171af5b14\") "
Oct 30 22:13:49.996505 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.996329    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-istio-data" pod "164e63e1-b309-4236-983a-2fd171af5b14" (UID: "164e63e1-b309-4236-983a-2fd171af5b14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:49.996627 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.996606    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/164e63e1-b309-4236-983a-2fd171af5b14-istiod-ca-cert" pod "164e63e1-b309-4236-983a-2fd171af5b14" (UID: "164e63e1-b309-4236-983a-2fd171af5b14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:49.996727 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.996707    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:49.996795 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.996734    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:49.996795 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.996749    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:49.996795 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.996763    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/164e63e1-b309-4236-983a-2fd171af5b14-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:49.997196 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.997165    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-credential-socket" pod "164e63e1-b309-4236-983a-2fd171af5b14" (UID: "164e63e1-b309-4236-983a-2fd171af5b14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:49.999478 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.999439    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/164e63e1-b309-4236-983a-2fd171af5b14-kube-api-access-v9l5v" pod "164e63e1-b309-4236-983a-2fd171af5b14" (UID: "164e63e1-b309-4236-983a-2fd171af5b14"). InnerVolumeSpecName "kube-api-access-v9l5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:49.999478 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.999451    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/164e63e1-b309-4236-983a-2fd171af5b14-istio-token" pod "164e63e1-b309-4236-983a-2fd171af5b14" (UID: "164e63e1-b309-4236-983a-2fd171af5b14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:49.999676 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:49.999655    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/164e63e1-b309-4236-983a-2fd171af5b14-istio-podinfo" pod "164e63e1-b309-4236-983a-2fd171af5b14" (UID: "164e63e1-b309-4236-983a-2fd171af5b14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:13:50.000976 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:50.000949    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-istio-envoy" pod "164e63e1-b309-4236-983a-2fd171af5b14" (UID: "164e63e1-b309-4236-983a-2fd171af5b14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:50.097961 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:50.097930    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/164e63e1-b309-4236-983a-2fd171af5b14-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:50.097961 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:50.097956    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:50.097961 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:50.097966    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9l5v\" (UniqueName: \"kubernetes.io/projected/164e63e1-b309-4236-983a-2fd171af5b14-kube-api-access-v9l5v\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:50.098194 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:50.097977    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/164e63e1-b309-4236-983a-2fd171af5b14-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:50.098194 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:50.097985    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/164e63e1-b309-4236-983a-2fd171af5b14-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:50.148564 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:50.148536    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"]
Oct 30 22:13:50.155780 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:50.155755    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"]
Oct 30 22:13:50.328461 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:50.328388    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb"
Oct 30 22:13:50.363533 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:50.363486    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b"
Oct 30 22:13:50.363533 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:50.363515    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b" event={"ID":"0fe89e6c-771c-4cb7-a4f5-54668eee2524","Type":"ContainerDied","Data":"52496c080b564a2108f16639d94ee6cec87d32355ff7f21196fca6d26abd88d3"}
Oct 30 22:13:50.363760 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:50.363569    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52496c080b564a2108f16639d94ee6cec87d32355ff7f21196fca6d26abd88d3"
Oct 30 22:13:50.366123 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:50.366070    5087 generic.go:356] "Generic (PLEG): container finished" podID="164e63e1-b309-4236-983a-2fd171af5b14" containerID="71bff2f05db10d65035ba080c86c05ff25df7ad6d47129088ed2b63d400c7f6d" exitCode=0
Oct 30 22:13:50.366294 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:50.366183    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"
Oct 30 22:13:50.366294 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:50.366275    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz" event={"ID":"164e63e1-b309-4236-983a-2fd171af5b14","Type":"ContainerDied","Data":"71bff2f05db10d65035ba080c86c05ff25df7ad6d47129088ed2b63d400c7f6d"}
Oct 30 22:13:50.366464 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:50.366322    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz" event={"ID":"164e63e1-b309-4236-983a-2fd171af5b14","Type":"ContainerDied","Data":"97709c773f8d361e3d7bb896c86ddcbdc84f84904f00dcf04135c50a8ae5035c"}
Oct 30 22:13:50.366464 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:50.366408    5087 scope.go:122] "RemoveContainer" containerID="71bff2f05db10d65035ba080c86c05ff25df7ad6d47129088ed2b63d400c7f6d"
Oct 30 22:13:50.379695 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:50.379679    5087 scope.go:122] "RemoveContainer" containerID="e1aeff8d7817ae5828f5734121dd9658c793153519f5374b39cc49105fd08bea"
Oct 30 22:13:50.392011 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:50.391992    5087 scope.go:122] "RemoveContainer" containerID="71bff2f05db10d65035ba080c86c05ff25df7ad6d47129088ed2b63d400c7f6d"
Oct 30 22:13:50.392265 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:13:50.392247    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71bff2f05db10d65035ba080c86c05ff25df7ad6d47129088ed2b63d400c7f6d\": container with ID starting with 71bff2f05db10d65035ba080c86c05ff25df7ad6d47129088ed2b63d400c7f6d not found: ID does not exist" containerID="71bff2f05db10d65035ba080c86c05ff25df7ad6d47129088ed2b63d400c7f6d"
Oct 30 22:13:50.392315 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:50.392276    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71bff2f05db10d65035ba080c86c05ff25df7ad6d47129088ed2b63d400c7f6d"} err="failed to get container status \"71bff2f05db10d65035ba080c86c05ff25df7ad6d47129088ed2b63d400c7f6d\": rpc error: code = NotFound desc = could not find container \"71bff2f05db10d65035ba080c86c05ff25df7ad6d47129088ed2b63d400c7f6d\": container with ID starting with 71bff2f05db10d65035ba080c86c05ff25df7ad6d47129088ed2b63d400c7f6d not found: ID does not exist"
Oct 30 22:13:50.392315 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:50.392297    5087 scope.go:122] "RemoveContainer" containerID="e1aeff8d7817ae5828f5734121dd9658c793153519f5374b39cc49105fd08bea"
Oct 30 22:13:50.392586 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:13:50.392567    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1aeff8d7817ae5828f5734121dd9658c793153519f5374b39cc49105fd08bea\": container with ID starting with e1aeff8d7817ae5828f5734121dd9658c793153519f5374b39cc49105fd08bea not found: ID does not exist" containerID="e1aeff8d7817ae5828f5734121dd9658c793153519f5374b39cc49105fd08bea"
Oct 30 22:13:50.392652 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:50.392595    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1aeff8d7817ae5828f5734121dd9658c793153519f5374b39cc49105fd08bea"} err="failed to get container status \"e1aeff8d7817ae5828f5734121dd9658c793153519f5374b39cc49105fd08bea\": rpc error: code = NotFound desc = could not find container \"e1aeff8d7817ae5828f5734121dd9658c793153519f5374b39cc49105fd08bea\": container with ID starting with e1aeff8d7817ae5828f5734121dd9658c793153519f5374b39cc49105fd08bea not found: ID does not exist"
Oct 30 22:13:50.552485 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:50.552446    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"]
Oct 30 22:13:50.555748 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:50.555727    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz"]
Oct 30 22:13:50.807818 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:50.807771    5087 patch_prober.go:29] interesting pod/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:13:50.808021 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:50.807847    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-307368-openshift-default-748d46b5cc-hzgkz" podUID="164e63e1-b309-4236-983a-2fd171af5b14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:13:51.736401 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:51.727337    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="164e63e1-b309-4236-983a-2fd171af5b14" path="/var/lib/kubelet/pods/164e63e1-b309-4236-983a-2fd171af5b14/volumes"
Oct 30 22:13:53.912411 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:53.912337    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-wqggm"]
Oct 30 22:13:53.963642 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:53.963604    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6df3ba0e-wqggm"]
Oct 30 22:13:55.709302 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:55.709270    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c57cf351-4009-4aaa-9ee6-c579ec319a7a" path="/var/lib/kubelet/pods/c57cf351-4009-4aaa-9ee6-c579ec319a7a/volumes"
Oct 30 22:13:56.333855 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:56.333818    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-557708-72cvv"]
Oct 30 22:13:56.334468 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:56.334446    5087 cpu_manager.go:414] "containerMap: removing container" podUID="164e63e1-b309-4236-983a-2fd171af5b14" containerName="istio-proxy"
Oct 30 22:13:56.334468 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:56.334469    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="164e63e1-b309-4236-983a-2fd171af5b14" containerName="istio-proxy"
Oct 30 22:13:56.334657 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:56.334506    5087 cpu_manager.go:414] "containerMap: removing container" podUID="164e63e1-b309-4236-983a-2fd171af5b14" containerName="istio-proxy"
Oct 30 22:13:56.334657 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:56.334513    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="164e63e1-b309-4236-983a-2fd171af5b14" containerName="istio-proxy"
Oct 30 22:13:56.334657 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:56.334539    5087 cpu_manager.go:414] "containerMap: removing container" podUID="0fe89e6c-771c-4cb7-a4f5-54668eee2524" containerName="revoke-keys"
Oct 30 22:13:56.334657 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:56.334546    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fe89e6c-771c-4cb7-a4f5-54668eee2524" containerName="revoke-keys"
Oct 30 22:13:56.334859 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:56.334700    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="164e63e1-b309-4236-983a-2fd171af5b14" containerName="istio-proxy"
Oct 30 22:13:56.334859 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:56.334715    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fe89e6c-771c-4cb7-a4f5-54668eee2524" containerName="revoke-keys"
Oct 30 22:13:56.339796 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:56.339775    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-557708-72cvv"
Oct 30 22:13:56.409617 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:56.409579    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-557708-72cvv"]
Oct 30 22:13:56.419980 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:56.419944    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbcxw\" (UniqueName: \"kubernetes.io/projected/17f4cbc0-822c-4137-8a9a-addf48bf49e4-kube-api-access-gbcxw\") pod \"maas-api-revoke-keys-e2e-worker-w6-557708-72cvv\" (UID: \"17f4cbc0-822c-4137-8a9a-addf48bf49e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-557708-72cvv"
Oct 30 22:13:56.420169 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:56.420029    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17f4cbc0-822c-4137-8a9a-addf48bf49e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-557708-72cvv\" (UID: \"17f4cbc0-822c-4137-8a9a-addf48bf49e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-557708-72cvv"
Oct 30 22:13:56.420169 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:56.420111    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17f4cbc0-822c-4137-8a9a-addf48bf49e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-557708-72cvv\" (UID: \"17f4cbc0-822c-4137-8a9a-addf48bf49e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-557708-72cvv"
Oct 30 22:13:56.438138 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:56.438108    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:13:56.454185 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:56.454155    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:13:56.536073 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:56.536033    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17f4cbc0-822c-4137-8a9a-addf48bf49e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-557708-72cvv\" (UID: \"17f4cbc0-822c-4137-8a9a-addf48bf49e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-557708-72cvv"
Oct 30 22:13:56.536290 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:56.536266    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbcxw\" (UniqueName: \"kubernetes.io/projected/17f4cbc0-822c-4137-8a9a-addf48bf49e4-kube-api-access-gbcxw\") pod \"maas-api-revoke-keys-e2e-worker-w6-557708-72cvv\" (UID: \"17f4cbc0-822c-4137-8a9a-addf48bf49e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-557708-72cvv"
Oct 30 22:13:56.536406 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:56.536312    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17f4cbc0-822c-4137-8a9a-addf48bf49e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-557708-72cvv\" (UID: \"17f4cbc0-822c-4137-8a9a-addf48bf49e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-557708-72cvv"
Oct 30 22:13:56.537674 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:56.537646    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17f4cbc0-822c-4137-8a9a-addf48bf49e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-557708-72cvv\" (UID: \"17f4cbc0-822c-4137-8a9a-addf48bf49e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-557708-72cvv"
Oct 30 22:13:56.540898 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:56.540763    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17f4cbc0-822c-4137-8a9a-addf48bf49e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-557708-72cvv\" (UID: \"17f4cbc0-822c-4137-8a9a-addf48bf49e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-557708-72cvv"
Oct 30 22:13:56.593973 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:56.593907    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbcxw\" (UniqueName: \"kubernetes.io/projected/17f4cbc0-822c-4137-8a9a-addf48bf49e4-kube-api-access-gbcxw\") pod \"maas-api-revoke-keys-e2e-worker-w6-557708-72cvv\" (UID: \"17f4cbc0-822c-4137-8a9a-addf48bf49e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-557708-72cvv"
Oct 30 22:13:56.664041 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:56.663997    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-557708-72cvv"
Oct 30 22:13:57.059593 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:57.059567    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-557708-72cvv"]
Oct 30 22:13:57.061251 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:13:57.061219    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f4cbc0_822c_4137_8a9a_addf48bf49e4.slice/crio-5787425b3c396a60156890a66adc4b131c2317a1a295fd0d737ab9f8c2a9396e WatchSource:0}: Error finding container 5787425b3c396a60156890a66adc4b131c2317a1a295fd0d737ab9f8c2a9396e: Status 404 returned error can't find the container with id 5787425b3c396a60156890a66adc4b131c2317a1a295fd0d737ab9f8c2a9396e
Oct 30 22:13:57.445280 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:57.445242    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-557708-72cvv" event={"ID":"17f4cbc0-822c-4137-8a9a-addf48bf49e4","Type":"ContainerStarted","Data":"5787425b3c396a60156890a66adc4b131c2317a1a295fd0d737ab9f8c2a9396e"}
Oct 30 22:13:58.455022 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:58.454940    5087 generic.go:356] "Generic (PLEG): container finished" podID="17f4cbc0-822c-4137-8a9a-addf48bf49e4" containerID="66f34129e1bbb0a909a0ef9e8d7700d7579f9bd5f90abd6e9d0487aca305f5bd" exitCode=0
Oct 30 22:13:58.455440 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:58.455024    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-557708-72cvv" event={"ID":"17f4cbc0-822c-4137-8a9a-addf48bf49e4","Type":"ContainerDied","Data":"66f34129e1bbb0a909a0ef9e8d7700d7579f9bd5f90abd6e9d0487aca305f5bd"}
Oct 30 22:13:59.617950 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:59.617926    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-557708-72cvv"
Oct 30 22:13:59.706439 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:59.706405    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17f4cbc0-822c-4137-8a9a-addf48bf49e4-kube-api-access-gbcxw\" (UniqueName: \"kubernetes.io/projected/17f4cbc0-822c-4137-8a9a-addf48bf49e4-kube-api-access-gbcxw\") pod \"17f4cbc0-822c-4137-8a9a-addf48bf49e4\" (UID: \"17f4cbc0-822c-4137-8a9a-addf48bf49e4\") "
Oct 30 22:13:59.706645 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:59.706545    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17f4cbc0-822c-4137-8a9a-addf48bf49e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17f4cbc0-822c-4137-8a9a-addf48bf49e4-maas-api-xks-ca\") pod \"17f4cbc0-822c-4137-8a9a-addf48bf49e4\" (UID: \"17f4cbc0-822c-4137-8a9a-addf48bf49e4\") "
Oct 30 22:13:59.706645 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:59.706608    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17f4cbc0-822c-4137-8a9a-addf48bf49e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17f4cbc0-822c-4137-8a9a-addf48bf49e4-maas-api-openshift-ca\") pod \"17f4cbc0-822c-4137-8a9a-addf48bf49e4\" (UID: \"17f4cbc0-822c-4137-8a9a-addf48bf49e4\") "
Oct 30 22:13:59.707454 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:59.707420    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17f4cbc0-822c-4137-8a9a-addf48bf49e4-maas-api-openshift-ca" pod "17f4cbc0-822c-4137-8a9a-addf48bf49e4" (UID: "17f4cbc0-822c-4137-8a9a-addf48bf49e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:59.709157 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:59.709084    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17f4cbc0-822c-4137-8a9a-addf48bf49e4-maas-api-xks-ca" pod "17f4cbc0-822c-4137-8a9a-addf48bf49e4" (UID: "17f4cbc0-822c-4137-8a9a-addf48bf49e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:13:59.709272 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:59.709226    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f4cbc0-822c-4137-8a9a-addf48bf49e4-kube-api-access-gbcxw" pod "17f4cbc0-822c-4137-8a9a-addf48bf49e4" (UID: "17f4cbc0-822c-4137-8a9a-addf48bf49e4"). InnerVolumeSpecName "kube-api-access-gbcxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:59.808446 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:59.808412    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbcxw\" (UniqueName: \"kubernetes.io/projected/17f4cbc0-822c-4137-8a9a-addf48bf49e4-kube-api-access-gbcxw\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:59.808446 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:59.808442    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17f4cbc0-822c-4137-8a9a-addf48bf49e4-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:13:59.808446 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:13:59.808452    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17f4cbc0-822c-4137-8a9a-addf48bf49e4-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:00.471076 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:00.471045    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-557708-72cvv"
Oct 30 22:14:00.471076 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:00.471063    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-557708-72cvv" event={"ID":"17f4cbc0-822c-4137-8a9a-addf48bf49e4","Type":"ContainerDied","Data":"5787425b3c396a60156890a66adc4b131c2317a1a295fd0d737ab9f8c2a9396e"}
Oct 30 22:14:00.471308 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:00.471112    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5787425b3c396a60156890a66adc4b131c2317a1a295fd0d737ab9f8c2a9396e"
Oct 30 22:14:00.745006 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:00.744974    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"]
Oct 30 22:14:00.756097 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:00.756069    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"]
Oct 30 22:14:05.170117 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.170066    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" containerID="cri-o://0b466c23455d3ba73bcb706f91b4ad6a5d0cb4c8f745256b45d4967a40e3e64f" gracePeriod=45
Oct 30 22:14:05.176650 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.176600    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main" containerID="cri-o://39f9e758e796c2c0a47624913a6194d955e4b7be579bb40a81695d84e8cc792d" gracePeriod=45
Oct 30 22:14:05.522653 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.522619    5087 generic.go:356] "Generic (PLEG): container finished" podID="b066538d-4d14-4228-b563-bce796a28d0b" containerID="39f9e758e796c2c0a47624913a6194d955e4b7be579bb40a81695d84e8cc792d" exitCode=0
Oct 30 22:14:05.522824 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.522795    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" event={"ID":"b066538d-4d14-4228-b563-bce796a28d0b","Type":"ContainerDied","Data":"39f9e758e796c2c0a47624913a6194d955e4b7be579bb40a81695d84e8cc792d"}
Oct 30 22:14:05.522903 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.522835    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" event={"ID":"b066538d-4d14-4228-b563-bce796a28d0b","Type":"ContainerDied","Data":"e31b0c31e0a5c99f58fcecf65b52e9e103f0f6730bbe396597a61489319d02f6"}
Oct 30 22:14:05.522903 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.522850    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e31b0c31e0a5c99f58fcecf65b52e9e103f0f6730bbe396597a61489319d02f6"
Oct 30 22:14:05.524506 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.524478    5087 generic.go:356] "Generic (PLEG): container finished" podID="cab3f652-e0fa-4351-b94a-271180e6c277" containerID="0b466c23455d3ba73bcb706f91b4ad6a5d0cb4c8f745256b45d4967a40e3e64f" exitCode=0
Oct 30 22:14:05.524640 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.524564    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" event={"ID":"cab3f652-e0fa-4351-b94a-271180e6c277","Type":"ContainerDied","Data":"0b466c23455d3ba73bcb706f91b4ad6a5d0cb4c8f745256b45d4967a40e3e64f"}
Oct 30 22:14:05.524640 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.524608    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" event={"ID":"cab3f652-e0fa-4351-b94a-271180e6c277","Type":"ContainerDied","Data":"412bcfae82b9355383bbb99e81c4650069ecdb7da4ab6bcae217ede471edb07f"}
Oct 30 22:14:05.524640 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.524622    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="412bcfae82b9355383bbb99e81c4650069ecdb7da4ab6bcae217ede471edb07f"
Oct 30 22:14:05.525981 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.525964    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 22:14:05.530284 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.530266    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 22:14:05.679815 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.679775    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cab3f652-e0fa-4351-b94a-271180e6c277-tls-certs\" (UniqueName: \"kubernetes.io/secret/cab3f652-e0fa-4351-b94a-271180e6c277-tls-certs\") pod \"cab3f652-e0fa-4351-b94a-271180e6c277\" (UID: \"cab3f652-e0fa-4351-b94a-271180e6c277\") "
Oct 30 22:14:05.680017 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.679875    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b066538d-4d14-4228-b563-bce796a28d0b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b066538d-4d14-4228-b563-bce796a28d0b-tmp-dir\") pod \"b066538d-4d14-4228-b563-bce796a28d0b\" (UID: \"b066538d-4d14-4228-b563-bce796a28d0b\") "
Oct 30 22:14:05.680017 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.679974    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b066538d-4d14-4228-b563-bce796a28d0b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b066538d-4d14-4228-b563-bce796a28d0b-model-cache\") pod \"b066538d-4d14-4228-b563-bce796a28d0b\" (UID: \"b066538d-4d14-4228-b563-bce796a28d0b\") "
Oct 30 22:14:05.680208 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.680176    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b066538d-4d14-4228-b563-bce796a28d0b-tmp-dir" pod "b066538d-4d14-4228-b563-bce796a28d0b" (UID: "b066538d-4d14-4228-b563-bce796a28d0b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:05.680306 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.680225    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b066538d-4d14-4228-b563-bce796a28d0b-model-cache" pod "b066538d-4d14-4228-b563-bce796a28d0b" (UID: "b066538d-4d14-4228-b563-bce796a28d0b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:05.680306 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.680007    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b066538d-4d14-4228-b563-bce796a28d0b-tls-certs\" (UniqueName: \"kubernetes.io/secret/b066538d-4d14-4228-b563-bce796a28d0b-tls-certs\") pod \"b066538d-4d14-4228-b563-bce796a28d0b\" (UID: \"b066538d-4d14-4228-b563-bce796a28d0b\") "
Oct 30 22:14:05.680449 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.680369    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab3f652-e0fa-4351-b94a-271180e6c277-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cab3f652-e0fa-4351-b94a-271180e6c277-tmp-dir\") pod \"cab3f652-e0fa-4351-b94a-271180e6c277\" (UID: \"cab3f652-e0fa-4351-b94a-271180e6c277\") "
Oct 30 22:14:05.680449 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.680417    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cab3f652-e0fa-4351-b94a-271180e6c277-kube-api-access-w8prf\" (UniqueName: \"kubernetes.io/projected/cab3f652-e0fa-4351-b94a-271180e6c277-kube-api-access-w8prf\") pod \"cab3f652-e0fa-4351-b94a-271180e6c277\" (UID: \"cab3f652-e0fa-4351-b94a-271180e6c277\") "
Oct 30 22:14:05.680565 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.680485    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b066538d-4d14-4228-b563-bce796a28d0b-home\" (UniqueName: \"kubernetes.io/empty-dir/b066538d-4d14-4228-b563-bce796a28d0b-home\") pod \"b066538d-4d14-4228-b563-bce796a28d0b\" (UID: \"b066538d-4d14-4228-b563-bce796a28d0b\") "
Oct 30 22:14:05.680721 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.680699    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab3f652-e0fa-4351-b94a-271180e6c277-tmp-dir" pod "cab3f652-e0fa-4351-b94a-271180e6c277" (UID: "cab3f652-e0fa-4351-b94a-271180e6c277"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:05.681020 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.680995    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab3f652-e0fa-4351-b94a-271180e6c277-home\" (UniqueName: \"kubernetes.io/empty-dir/cab3f652-e0fa-4351-b94a-271180e6c277-home\") pod \"cab3f652-e0fa-4351-b94a-271180e6c277\" (UID: \"cab3f652-e0fa-4351-b94a-271180e6c277\") "
Oct 30 22:14:05.681122 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.681095    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab3f652-e0fa-4351-b94a-271180e6c277-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cab3f652-e0fa-4351-b94a-271180e6c277-model-cache\") pod \"cab3f652-e0fa-4351-b94a-271180e6c277\" (UID: \"cab3f652-e0fa-4351-b94a-271180e6c277\") "
Oct 30 22:14:05.681187 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.681144    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b066538d-4d14-4228-b563-bce796a28d0b-kube-api-access-8gh7z\" (UniqueName: \"kubernetes.io/projected/b066538d-4d14-4228-b563-bce796a28d0b-kube-api-access-8gh7z\") pod \"b066538d-4d14-4228-b563-bce796a28d0b\" (UID: \"b066538d-4d14-4228-b563-bce796a28d0b\") "
Oct 30 22:14:05.681250 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.681232    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b066538d-4d14-4228-b563-bce796a28d0b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b066538d-4d14-4228-b563-bce796a28d0b-dshm\") pod \"b066538d-4d14-4228-b563-bce796a28d0b\" (UID: \"b066538d-4d14-4228-b563-bce796a28d0b\") "
Oct 30 22:14:05.681311 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.681258    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cab3f652-e0fa-4351-b94a-271180e6c277-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cab3f652-e0fa-4351-b94a-271180e6c277-dshm\") pod \"cab3f652-e0fa-4351-b94a-271180e6c277\" (UID: \"cab3f652-e0fa-4351-b94a-271180e6c277\") "
Oct 30 22:14:05.681651 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.681622    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab3f652-e0fa-4351-b94a-271180e6c277-model-cache" pod "cab3f652-e0fa-4351-b94a-271180e6c277" (UID: "cab3f652-e0fa-4351-b94a-271180e6c277"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:05.682003 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.681979    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b066538d-4d14-4228-b563-bce796a28d0b-home" pod "b066538d-4d14-4228-b563-bce796a28d0b" (UID: "b066538d-4d14-4228-b563-bce796a28d0b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:05.682268 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.682243    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab3f652-e0fa-4351-b94a-271180e6c277-home" pod "cab3f652-e0fa-4351-b94a-271180e6c277" (UID: "cab3f652-e0fa-4351-b94a-271180e6c277"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:05.684198 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.683720    5087 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b066538d-4d14-4228-b563-bce796a28d0b-tmp-dir\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:05.684198 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.683751    5087 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b066538d-4d14-4228-b563-bce796a28d0b-model-cache\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:05.684198 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.683767    5087 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cab3f652-e0fa-4351-b94a-271180e6c277-tmp-dir\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:05.684198 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.683782    5087 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b066538d-4d14-4228-b563-bce796a28d0b-home\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:05.684198 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.683795    5087 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cab3f652-e0fa-4351-b94a-271180e6c277-home\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:05.684198 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.683808    5087 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cab3f652-e0fa-4351-b94a-271180e6c277-model-cache\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:05.685947 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.685916    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab3f652-e0fa-4351-b94a-271180e6c277-kube-api-access-w8prf" pod "cab3f652-e0fa-4351-b94a-271180e6c277" (UID: "cab3f652-e0fa-4351-b94a-271180e6c277"). InnerVolumeSpecName "kube-api-access-w8prf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:05.686166 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.686139    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cab3f652-e0fa-4351-b94a-271180e6c277-tls-certs" pod "cab3f652-e0fa-4351-b94a-271180e6c277" (UID: "cab3f652-e0fa-4351-b94a-271180e6c277"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:05.686275 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.686144    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b066538d-4d14-4228-b563-bce796a28d0b-tls-certs" pod "b066538d-4d14-4228-b563-bce796a28d0b" (UID: "b066538d-4d14-4228-b563-bce796a28d0b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:05.686341 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.686295    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cab3f652-e0fa-4351-b94a-271180e6c277-dshm" pod "cab3f652-e0fa-4351-b94a-271180e6c277" (UID: "cab3f652-e0fa-4351-b94a-271180e6c277"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:05.686625 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.686594    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b066538d-4d14-4228-b563-bce796a28d0b-dshm" pod "b066538d-4d14-4228-b563-bce796a28d0b" (UID: "b066538d-4d14-4228-b563-bce796a28d0b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:05.686625 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.686616    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b066538d-4d14-4228-b563-bce796a28d0b-kube-api-access-8gh7z" pod "b066538d-4d14-4228-b563-bce796a28d0b" (UID: "b066538d-4d14-4228-b563-bce796a28d0b"). InnerVolumeSpecName "kube-api-access-8gh7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:05.786231 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.784895    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8prf\" (UniqueName: \"kubernetes.io/projected/cab3f652-e0fa-4351-b94a-271180e6c277-kube-api-access-w8prf\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:05.786231 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.784931    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gh7z\" (UniqueName: \"kubernetes.io/projected/b066538d-4d14-4228-b563-bce796a28d0b-kube-api-access-8gh7z\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:05.786231 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.784945    5087 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b066538d-4d14-4228-b563-bce796a28d0b-dshm\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:05.786231 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.784959    5087 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cab3f652-e0fa-4351-b94a-271180e6c277-dshm\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:05.786231 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.784971    5087 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cab3f652-e0fa-4351-b94a-271180e6c277-tls-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:05.786231 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:05.784983    5087 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b066538d-4d14-4228-b563-bce796a28d0b-tls-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:06.409616 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:06.409574    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main" probeResult="failure" output="Get \"https://10.132.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:06.412739 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:06.412711    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:14:06.533334 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:06.533305    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"
Oct 30 22:14:06.533580 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:06.533305    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"
Oct 30 22:14:06.753091 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:06.753051    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"]
Oct 30 22:14:06.757816 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:06.757789    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c3e2d8/simulator-w4-c3e2d8-kserve-7bbd8f768d-hvcvj"]
Oct 30 22:14:06.863001 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:06.862916    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"]
Oct 30 22:14:06.868322 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:06.868290    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-c3e2d8/premium-w4-c3e2d8-kserve-8757b9cd5-dxcsf"]
Oct 30 22:14:07.709189 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:07.709158    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b066538d-4d14-4228-b563-bce796a28d0b" path="/var/lib/kubelet/pods/b066538d-4d14-4228-b563-bce796a28d0b/volumes"
Oct 30 22:14:07.709633 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:07.709614    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" path="/var/lib/kubelet/pods/cab3f652-e0fa-4351-b94a-271180e6c277/volumes"
Oct 30 22:14:13.015268 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.015235    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4ea03cd-zzd66"]
Oct 30 22:14:13.015801 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.015785    5087 cpu_manager.go:414] "containerMap: removing container" podUID="17f4cbc0-822c-4137-8a9a-addf48bf49e4" containerName="revoke-keys"
Oct 30 22:14:13.015858 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.015801    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17f4cbc0-822c-4137-8a9a-addf48bf49e4" containerName="revoke-keys"
Oct 30 22:14:13.015858 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.015835    5087 cpu_manager.go:414] "containerMap: removing container" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main"
Oct 30 22:14:13.015858 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.015840    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main"
Oct 30 22:14:13.015955 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.015860    5087 cpu_manager.go:414] "containerMap: removing container" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main"
Oct 30 22:14:13.015955 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.015865    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main"
Oct 30 22:14:13.015955 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.015936    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="164e63e1-b309-4236-983a-2fd171af5b14" containerName="istio-proxy"
Oct 30 22:14:13.015955 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.015946    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="cab3f652-e0fa-4351-b94a-271180e6c277" containerName="main"
Oct 30 22:14:13.016076 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.015957    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="b066538d-4d14-4228-b563-bce796a28d0b" containerName="main"
Oct 30 22:14:13.016076 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.015964    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="17f4cbc0-822c-4137-8a9a-addf48bf49e4" containerName="revoke-keys"
Oct 30 22:14:13.020685 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.020669    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ea03cd-zzd66"
Oct 30 22:14:13.077988 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.077960    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4ea03cd-zzd66"]
Oct 30 22:14:13.088088 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.088057    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acf0442b-7e18-44ad-b993-65ca686d6013-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4ea03cd-zzd66\" (UID: \"acf0442b-7e18-44ad-b993-65ca686d6013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ea03cd-zzd66"
Oct 30 22:14:13.088225 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.088194    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acf0442b-7e18-44ad-b993-65ca686d6013-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4ea03cd-zzd66\" (UID: \"acf0442b-7e18-44ad-b993-65ca686d6013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ea03cd-zzd66"
Oct 30 22:14:13.088276 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.088259    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rkfz\" (UniqueName: \"kubernetes.io/projected/acf0442b-7e18-44ad-b993-65ca686d6013-kube-api-access-6rkfz\") pod \"maas-api-revoke-sub-e4ea03cd-zzd66\" (UID: \"acf0442b-7e18-44ad-b993-65ca686d6013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ea03cd-zzd66"
Oct 30 22:14:13.123494 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.123466    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:14:13.123494 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.123466    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:14:13.190109 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.190067    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acf0442b-7e18-44ad-b993-65ca686d6013-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4ea03cd-zzd66\" (UID: \"acf0442b-7e18-44ad-b993-65ca686d6013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ea03cd-zzd66"
Oct 30 22:14:13.190307 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.190136    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acf0442b-7e18-44ad-b993-65ca686d6013-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4ea03cd-zzd66\" (UID: \"acf0442b-7e18-44ad-b993-65ca686d6013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ea03cd-zzd66"
Oct 30 22:14:13.190307 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.190195    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rkfz\" (UniqueName: \"kubernetes.io/projected/acf0442b-7e18-44ad-b993-65ca686d6013-kube-api-access-6rkfz\") pod \"maas-api-revoke-sub-e4ea03cd-zzd66\" (UID: \"acf0442b-7e18-44ad-b993-65ca686d6013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ea03cd-zzd66"
Oct 30 22:14:13.190768 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.190747    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acf0442b-7e18-44ad-b993-65ca686d6013-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4ea03cd-zzd66\" (UID: \"acf0442b-7e18-44ad-b993-65ca686d6013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ea03cd-zzd66"
Oct 30 22:14:13.192594 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.192571    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acf0442b-7e18-44ad-b993-65ca686d6013-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4ea03cd-zzd66\" (UID: \"acf0442b-7e18-44ad-b993-65ca686d6013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ea03cd-zzd66"
Oct 30 22:14:13.257457 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.257424    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rkfz\" (UniqueName: \"kubernetes.io/projected/acf0442b-7e18-44ad-b993-65ca686d6013-kube-api-access-6rkfz\") pod \"maas-api-revoke-sub-e4ea03cd-zzd66\" (UID: \"acf0442b-7e18-44ad-b993-65ca686d6013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ea03cd-zzd66"
Oct 30 22:14:13.265430 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.265332    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dc4c98b-b52m5"]
Oct 30 22:14:13.272250 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.272218    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4c98b-b52m5"
Oct 30 22:14:13.291743 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.291716    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f9b70cb-bfbf-40cd-beab-1860e8d63d7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dc4c98b-b52m5\" (UID: \"4f9b70cb-bfbf-40cd-beab-1860e8d63d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4c98b-b52m5"
Oct 30 22:14:13.291885 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.291780    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqt5x\" (UniqueName: \"kubernetes.io/projected/4f9b70cb-bfbf-40cd-beab-1860e8d63d7a-kube-api-access-kqt5x\") pod \"maas-api-revoke-sub-1dc4c98b-b52m5\" (UID: \"4f9b70cb-bfbf-40cd-beab-1860e8d63d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4c98b-b52m5"
Oct 30 22:14:13.291984 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.291962    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f9b70cb-bfbf-40cd-beab-1860e8d63d7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dc4c98b-b52m5\" (UID: \"4f9b70cb-bfbf-40cd-beab-1860e8d63d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4c98b-b52m5"
Oct 30 22:14:13.323754 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.323725    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dc4c98b-b52m5"]
Oct 30 22:14:13.334331 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.334305    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ea03cd-zzd66"
Oct 30 22:14:13.393603 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.393519    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f9b70cb-bfbf-40cd-beab-1860e8d63d7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dc4c98b-b52m5\" (UID: \"4f9b70cb-bfbf-40cd-beab-1860e8d63d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4c98b-b52m5"
Oct 30 22:14:13.396397 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.393636    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqt5x\" (UniqueName: \"kubernetes.io/projected/4f9b70cb-bfbf-40cd-beab-1860e8d63d7a-kube-api-access-kqt5x\") pod \"maas-api-revoke-sub-1dc4c98b-b52m5\" (UID: \"4f9b70cb-bfbf-40cd-beab-1860e8d63d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4c98b-b52m5"
Oct 30 22:14:13.396397 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.393815    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f9b70cb-bfbf-40cd-beab-1860e8d63d7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dc4c98b-b52m5\" (UID: \"4f9b70cb-bfbf-40cd-beab-1860e8d63d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4c98b-b52m5"
Oct 30 22:14:13.396397 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.394692    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f9b70cb-bfbf-40cd-beab-1860e8d63d7a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1dc4c98b-b52m5\" (UID: \"4f9b70cb-bfbf-40cd-beab-1860e8d63d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4c98b-b52m5"
Oct 30 22:14:13.401472 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.400527    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f9b70cb-bfbf-40cd-beab-1860e8d63d7a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1dc4c98b-b52m5\" (UID: \"4f9b70cb-bfbf-40cd-beab-1860e8d63d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4c98b-b52m5"
Oct 30 22:14:13.452222 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.452188    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqt5x\" (UniqueName: \"kubernetes.io/projected/4f9b70cb-bfbf-40cd-beab-1860e8d63d7a-kube-api-access-kqt5x\") pod \"maas-api-revoke-sub-1dc4c98b-b52m5\" (UID: \"4f9b70cb-bfbf-40cd-beab-1860e8d63d7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4c98b-b52m5"
Oct 30 22:14:13.586913 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.586827    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4c98b-b52m5"
Oct 30 22:14:13.721643 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.721620    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4ea03cd-zzd66"]
Oct 30 22:14:13.723067 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:14:13.723035    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacf0442b_7e18_44ad_b993_65ca686d6013.slice/crio-1eee06f8927ba14d8f539159984d0552479c4657a5308a45699572314a24947e WatchSource:0}: Error finding container 1eee06f8927ba14d8f539159984d0552479c4657a5308a45699572314a24947e: Status 404 returned error can't find the container with id 1eee06f8927ba14d8f539159984d0552479c4657a5308a45699572314a24947e
Oct 30 22:14:13.962545 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:13.962521    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dc4c98b-b52m5"]
Oct 30 22:14:13.963273 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:14:13.963253    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f9b70cb_bfbf_40cd_beab_1860e8d63d7a.slice/crio-4f180f659e0bc5209afdb45fd8a1d941e84d4cd59306fffc06987e4e07e685fd WatchSource:0}: Error finding container 4f180f659e0bc5209afdb45fd8a1d941e84d4cd59306fffc06987e4e07e685fd: Status 404 returned error can't find the container with id 4f180f659e0bc5209afdb45fd8a1d941e84d4cd59306fffc06987e4e07e685fd
Oct 30 22:14:14.601724 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:14.601623    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4c98b-b52m5" event={"ID":"4f9b70cb-bfbf-40cd-beab-1860e8d63d7a","Type":"ContainerStarted","Data":"778060b48a4e289187ebfa73e8d77c0546f4118d80f8f9c1d20227b53aed4f64"}
Oct 30 22:14:14.601724 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:14.601689    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4c98b-b52m5" event={"ID":"4f9b70cb-bfbf-40cd-beab-1860e8d63d7a","Type":"ContainerStarted","Data":"4f180f659e0bc5209afdb45fd8a1d941e84d4cd59306fffc06987e4e07e685fd"}
Oct 30 22:14:14.603310 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:14.603279    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ea03cd-zzd66" event={"ID":"acf0442b-7e18-44ad-b993-65ca686d6013","Type":"ContainerStarted","Data":"6dcd021446ef0739cb2c767c0feb2964e02fea0201b1aa82942e28c4494d864a"}
Oct 30 22:14:14.603444 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:14.603327    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ea03cd-zzd66" event={"ID":"acf0442b-7e18-44ad-b993-65ca686d6013","Type":"ContainerStarted","Data":"1eee06f8927ba14d8f539159984d0552479c4657a5308a45699572314a24947e"}
Oct 30 22:14:14.716142 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:14.716094    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4c98b-b52m5" podStartSLOduration=1.716080533 podStartE2EDuration="1.716080533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:14.714320326 +0000 UTC m=+2449.188824306" watchObservedRunningTime="2026-10-30 22:14:14.716080533 +0000 UTC m=+2449.190584520"
Oct 30 22:14:14.832240 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:14.832181    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ea03cd-zzd66" podStartSLOduration=2.83216251 podStartE2EDuration="2.83216251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:14.831085723 +0000 UTC m=+2449.305589704" watchObservedRunningTime="2026-10-30 22:14:14.83216251 +0000 UTC m=+2449.306666492"
Oct 30 22:14:15.611534 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:15.611498    5087 generic.go:356] "Generic (PLEG): container finished" podID="4f9b70cb-bfbf-40cd-beab-1860e8d63d7a" containerID="778060b48a4e289187ebfa73e8d77c0546f4118d80f8f9c1d20227b53aed4f64" exitCode=0
Oct 30 22:14:15.612052 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:15.611586    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4c98b-b52m5" event={"ID":"4f9b70cb-bfbf-40cd-beab-1860e8d63d7a","Type":"ContainerDied","Data":"778060b48a4e289187ebfa73e8d77c0546f4118d80f8f9c1d20227b53aed4f64"}
Oct 30 22:14:15.613232 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:15.613205    5087 generic.go:356] "Generic (PLEG): container finished" podID="acf0442b-7e18-44ad-b993-65ca686d6013" containerID="6dcd021446ef0739cb2c767c0feb2964e02fea0201b1aa82942e28c4494d864a" exitCode=0
Oct 30 22:14:15.613415 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:15.613280    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ea03cd-zzd66" event={"ID":"acf0442b-7e18-44ad-b993-65ca686d6013","Type":"ContainerDied","Data":"6dcd021446ef0739cb2c767c0feb2964e02fea0201b1aa82942e28c4494d864a"}
Oct 30 22:14:15.767282 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:15.767244    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerName="main" containerID="cri-o://07a6d7d8fc0834b6f69dec73f8813ac77f2c9e7a297c750c644dbdb75e4b0f18" gracePeriod=45
Oct 30 22:14:15.777114 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:15.777086    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerName="main" containerID="cri-o://d9536199e5a33bbdc75f3aeca78b92be08e5befec60873054fcbd748aae142d5" gracePeriod=45
Oct 30 22:14:16.097588 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.097562    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 22:14:16.099630 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.099612    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 22:14:16.122244 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.122225    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/699a4ca0-a268-4109-a8f8-dde6bf132689-tls-certs\" (UniqueName: \"kubernetes.io/secret/699a4ca0-a268-4109-a8f8-dde6bf132689-tls-certs\") pod \"699a4ca0-a268-4109-a8f8-dde6bf132689\" (UID: \"699a4ca0-a268-4109-a8f8-dde6bf132689\") "
Oct 30 22:14:16.122462 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.122294    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/699a4ca0-a268-4109-a8f8-dde6bf132689-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/699a4ca0-a268-4109-a8f8-dde6bf132689-tmp-dir\") pod \"699a4ca0-a268-4109-a8f8-dde6bf132689\" (UID: \"699a4ca0-a268-4109-a8f8-dde6bf132689\") "
Oct 30 22:14:16.122462 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.122338    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/699a4ca0-a268-4109-a8f8-dde6bf132689-dshm\" (UniqueName: \"kubernetes.io/empty-dir/699a4ca0-a268-4109-a8f8-dde6bf132689-dshm\") pod \"699a4ca0-a268-4109-a8f8-dde6bf132689\" (UID: \"699a4ca0-a268-4109-a8f8-dde6bf132689\") "
Oct 30 22:14:16.122462 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.122404    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-tmp-dir\") pod \"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf\" (UID: \"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf\") "
Oct 30 22:14:16.122462 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.122422    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-tls-certs\" (UniqueName: \"kubernetes.io/secret/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-tls-certs\") pod \"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf\" (UID: \"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf\") "
Oct 30 22:14:16.122462 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.122451    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-home\" (UniqueName: \"kubernetes.io/empty-dir/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-home\") pod \"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf\" (UID: \"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf\") "
Oct 30 22:14:16.122803 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.122486    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-dshm\") pod \"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf\" (UID: \"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf\") "
Oct 30 22:14:16.122803 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.122520    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/699a4ca0-a268-4109-a8f8-dde6bf132689-kube-api-access-lhp7n\" (UniqueName: \"kubernetes.io/projected/699a4ca0-a268-4109-a8f8-dde6bf132689-kube-api-access-lhp7n\") pod \"699a4ca0-a268-4109-a8f8-dde6bf132689\" (UID: \"699a4ca0-a268-4109-a8f8-dde6bf132689\") "
Oct 30 22:14:16.122803 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.122554    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-model-cache\") pod \"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf\" (UID: \"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf\") "
Oct 30 22:14:16.122803 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.122580    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/699a4ca0-a268-4109-a8f8-dde6bf132689-tmp-dir" pod "699a4ca0-a268-4109-a8f8-dde6bf132689" (UID: "699a4ca0-a268-4109-a8f8-dde6bf132689"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:16.122803 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.122596    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-kube-api-access-8dqhp\" (UniqueName: \"kubernetes.io/projected/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-kube-api-access-8dqhp\") pod \"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf\" (UID: \"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf\") "
Oct 30 22:14:16.122803 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.122693    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-tmp-dir" pod "b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" (UID: "b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:16.122803 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.122722    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/699a4ca0-a268-4109-a8f8-dde6bf132689-home\" (UniqueName: \"kubernetes.io/empty-dir/699a4ca0-a268-4109-a8f8-dde6bf132689-home\") pod \"699a4ca0-a268-4109-a8f8-dde6bf132689\" (UID: \"699a4ca0-a268-4109-a8f8-dde6bf132689\") "
Oct 30 22:14:16.122803 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.122753    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/699a4ca0-a268-4109-a8f8-dde6bf132689-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/699a4ca0-a268-4109-a8f8-dde6bf132689-model-cache\") pod \"699a4ca0-a268-4109-a8f8-dde6bf132689\" (UID: \"699a4ca0-a268-4109-a8f8-dde6bf132689\") "
Oct 30 22:14:16.123230 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.122885    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-model-cache" pod "b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" (UID: "b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:16.123289 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.123221    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-home" pod "b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" (UID: "b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:16.123289 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.123272    5087 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-tmp-dir\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:16.123433 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.123292    5087 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-model-cache\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:16.123433 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.123306    5087 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/699a4ca0-a268-4109-a8f8-dde6bf132689-tmp-dir\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:16.123545 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.123451    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/699a4ca0-a268-4109-a8f8-dde6bf132689-home" pod "699a4ca0-a268-4109-a8f8-dde6bf132689" (UID: "699a4ca0-a268-4109-a8f8-dde6bf132689"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:16.123762 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.123678    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/699a4ca0-a268-4109-a8f8-dde6bf132689-model-cache" pod "699a4ca0-a268-4109-a8f8-dde6bf132689" (UID: "699a4ca0-a268-4109-a8f8-dde6bf132689"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:16.124859 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.124827    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/699a4ca0-a268-4109-a8f8-dde6bf132689-tls-certs" pod "699a4ca0-a268-4109-a8f8-dde6bf132689" (UID: "699a4ca0-a268-4109-a8f8-dde6bf132689"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:16.125200 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.125179    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-kube-api-access-8dqhp" pod "b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" (UID: "b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf"). InnerVolumeSpecName "kube-api-access-8dqhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:16.125283 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.125233    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/699a4ca0-a268-4109-a8f8-dde6bf132689-kube-api-access-lhp7n" pod "699a4ca0-a268-4109-a8f8-dde6bf132689" (UID: "699a4ca0-a268-4109-a8f8-dde6bf132689"). InnerVolumeSpecName "kube-api-access-lhp7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:16.125283 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.125252    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/699a4ca0-a268-4109-a8f8-dde6bf132689-dshm" pod "699a4ca0-a268-4109-a8f8-dde6bf132689" (UID: "699a4ca0-a268-4109-a8f8-dde6bf132689"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:16.125697 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.125677    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-tls-certs" pod "b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" (UID: "b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:16.125957 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.125943    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-dshm" pod "b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" (UID: "b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:16.224326 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.224242    5087 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/699a4ca0-a268-4109-a8f8-dde6bf132689-dshm\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:16.224326 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.224279    5087 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-tls-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:16.224326 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.224293    5087 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-home\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:16.224326 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.224306    5087 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-dshm\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:16.224326 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.224318    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhp7n\" (UniqueName: \"kubernetes.io/projected/699a4ca0-a268-4109-a8f8-dde6bf132689-kube-api-access-lhp7n\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:16.224770 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.224331    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dqhp\" (UniqueName: \"kubernetes.io/projected/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf-kube-api-access-8dqhp\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:16.224770 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.224364    5087 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/699a4ca0-a268-4109-a8f8-dde6bf132689-home\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:16.224770 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.224378    5087 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/699a4ca0-a268-4109-a8f8-dde6bf132689-model-cache\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:16.224770 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.224392    5087 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/699a4ca0-a268-4109-a8f8-dde6bf132689-tls-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:16.621606 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.621573    5087 generic.go:356] "Generic (PLEG): container finished" podID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerID="d9536199e5a33bbdc75f3aeca78b92be08e5befec60873054fcbd748aae142d5" exitCode=0
Oct 30 22:14:16.622030 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.621622    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"
Oct 30 22:14:16.622030 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.621649    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" event={"ID":"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf","Type":"ContainerDied","Data":"d9536199e5a33bbdc75f3aeca78b92be08e5befec60873054fcbd748aae142d5"}
Oct 30 22:14:16.622030 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.621690    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7" event={"ID":"b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf","Type":"ContainerDied","Data":"a99c8eba11020044b14e9d68bb017e04070612308449069560e7597d73b7d89e"}
Oct 30 22:14:16.622030 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.621711    5087 scope.go:122] "RemoveContainer" containerID="d9536199e5a33bbdc75f3aeca78b92be08e5befec60873054fcbd748aae142d5"
Oct 30 22:14:16.623197 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.623173    5087 generic.go:356] "Generic (PLEG): container finished" podID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerID="07a6d7d8fc0834b6f69dec73f8813ac77f2c9e7a297c750c644dbdb75e4b0f18" exitCode=0
Oct 30 22:14:16.623333 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.623216    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"
Oct 30 22:14:16.623333 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.623239    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" event={"ID":"699a4ca0-a268-4109-a8f8-dde6bf132689","Type":"ContainerDied","Data":"07a6d7d8fc0834b6f69dec73f8813ac77f2c9e7a297c750c644dbdb75e4b0f18"}
Oct 30 22:14:16.623333 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.623271    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc" event={"ID":"699a4ca0-a268-4109-a8f8-dde6bf132689","Type":"ContainerDied","Data":"628a197c0fea52dcd7700addc89ef26df239fe285e78a25604acaa7a9ba6d7b2"}
Oct 30 22:14:16.635145 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.635117    5087 scope.go:122] "RemoveContainer" containerID="d9536199e5a33bbdc75f3aeca78b92be08e5befec60873054fcbd748aae142d5"
Oct 30 22:14:16.635598 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:14:16.635494    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9536199e5a33bbdc75f3aeca78b92be08e5befec60873054fcbd748aae142d5\": container with ID starting with d9536199e5a33bbdc75f3aeca78b92be08e5befec60873054fcbd748aae142d5 not found: ID does not exist" containerID="d9536199e5a33bbdc75f3aeca78b92be08e5befec60873054fcbd748aae142d5"
Oct 30 22:14:16.635598 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.635529    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9536199e5a33bbdc75f3aeca78b92be08e5befec60873054fcbd748aae142d5"} err="failed to get container status \"d9536199e5a33bbdc75f3aeca78b92be08e5befec60873054fcbd748aae142d5\": rpc error: code = NotFound desc = could not find container \"d9536199e5a33bbdc75f3aeca78b92be08e5befec60873054fcbd748aae142d5\": container with ID starting with d9536199e5a33bbdc75f3aeca78b92be08e5befec60873054fcbd748aae142d5 not found: ID does not exist"
Oct 30 22:14:16.635598 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.635553    5087 scope.go:122] "RemoveContainer" containerID="07a6d7d8fc0834b6f69dec73f8813ac77f2c9e7a297c750c644dbdb75e4b0f18"
Oct 30 22:14:16.646972 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.646945    5087 scope.go:122] "RemoveContainer" containerID="07a6d7d8fc0834b6f69dec73f8813ac77f2c9e7a297c750c644dbdb75e4b0f18"
Oct 30 22:14:16.647280 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:14:16.647253    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07a6d7d8fc0834b6f69dec73f8813ac77f2c9e7a297c750c644dbdb75e4b0f18\": container with ID starting with 07a6d7d8fc0834b6f69dec73f8813ac77f2c9e7a297c750c644dbdb75e4b0f18 not found: ID does not exist" containerID="07a6d7d8fc0834b6f69dec73f8813ac77f2c9e7a297c750c644dbdb75e4b0f18"
Oct 30 22:14:16.647388 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.647289    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07a6d7d8fc0834b6f69dec73f8813ac77f2c9e7a297c750c644dbdb75e4b0f18"} err="failed to get container status \"07a6d7d8fc0834b6f69dec73f8813ac77f2c9e7a297c750c644dbdb75e4b0f18\": rpc error: code = NotFound desc = could not find container \"07a6d7d8fc0834b6f69dec73f8813ac77f2c9e7a297c750c644dbdb75e4b0f18\": container with ID starting with 07a6d7d8fc0834b6f69dec73f8813ac77f2c9e7a297c750c644dbdb75e4b0f18 not found: ID does not exist"
Oct 30 22:14:16.794755 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.793078    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"]
Oct 30 22:14:16.834717 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.834693    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4c98b-b52m5"
Oct 30 22:14:16.835776 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.835755    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-557708/simulator-w6-557708-kserve-54b6d54c8f-8nsj7"]
Oct 30 22:14:16.837363 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.837329    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ea03cd-zzd66"
Oct 30 22:14:16.904554 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.904491    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"]
Oct 30 22:14:16.908606 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.908582    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-557708/premium-w6-557708-kserve-5575f79844-cr9xc"]
Oct 30 22:14:16.932513 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.932487    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acf0442b-7e18-44ad-b993-65ca686d6013-kube-api-access-6rkfz\" (UniqueName: \"kubernetes.io/projected/acf0442b-7e18-44ad-b993-65ca686d6013-kube-api-access-6rkfz\") pod \"acf0442b-7e18-44ad-b993-65ca686d6013\" (UID: \"acf0442b-7e18-44ad-b993-65ca686d6013\") "
Oct 30 22:14:16.932615 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.932554    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f9b70cb-bfbf-40cd-beab-1860e8d63d7a-kube-api-access-kqt5x\" (UniqueName: \"kubernetes.io/projected/4f9b70cb-bfbf-40cd-beab-1860e8d63d7a-kube-api-access-kqt5x\") pod \"4f9b70cb-bfbf-40cd-beab-1860e8d63d7a\" (UID: \"4f9b70cb-bfbf-40cd-beab-1860e8d63d7a\") "
Oct 30 22:14:16.932615 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.932609    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f9b70cb-bfbf-40cd-beab-1860e8d63d7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f9b70cb-bfbf-40cd-beab-1860e8d63d7a-maas-api-xks-ca\") pod \"4f9b70cb-bfbf-40cd-beab-1860e8d63d7a\" (UID: \"4f9b70cb-bfbf-40cd-beab-1860e8d63d7a\") "
Oct 30 22:14:16.932750 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.932733    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acf0442b-7e18-44ad-b993-65ca686d6013-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acf0442b-7e18-44ad-b993-65ca686d6013-maas-api-xks-ca\") pod \"acf0442b-7e18-44ad-b993-65ca686d6013\" (UID: \"acf0442b-7e18-44ad-b993-65ca686d6013\") "
Oct 30 22:14:16.932802 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.932790    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f9b70cb-bfbf-40cd-beab-1860e8d63d7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f9b70cb-bfbf-40cd-beab-1860e8d63d7a-maas-api-openshift-ca\") pod \"4f9b70cb-bfbf-40cd-beab-1860e8d63d7a\" (UID: \"4f9b70cb-bfbf-40cd-beab-1860e8d63d7a\") "
Oct 30 22:14:16.933119 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.933092    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acf0442b-7e18-44ad-b993-65ca686d6013-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acf0442b-7e18-44ad-b993-65ca686d6013-maas-api-openshift-ca\") pod \"acf0442b-7e18-44ad-b993-65ca686d6013\" (UID: \"acf0442b-7e18-44ad-b993-65ca686d6013\") "
Oct 30 22:14:16.933228 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.933118    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f9b70cb-bfbf-40cd-beab-1860e8d63d7a-maas-api-openshift-ca" pod "4f9b70cb-bfbf-40cd-beab-1860e8d63d7a" (UID: "4f9b70cb-bfbf-40cd-beab-1860e8d63d7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:16.933508 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.933482    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acf0442b-7e18-44ad-b993-65ca686d6013-maas-api-openshift-ca" pod "acf0442b-7e18-44ad-b993-65ca686d6013" (UID: "acf0442b-7e18-44ad-b993-65ca686d6013"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:16.933889 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.933867    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f9b70cb-bfbf-40cd-beab-1860e8d63d7a-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:16.933939 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.933898    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acf0442b-7e18-44ad-b993-65ca686d6013-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:16.934965 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.934939    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f9b70cb-bfbf-40cd-beab-1860e8d63d7a-maas-api-xks-ca" pod "4f9b70cb-bfbf-40cd-beab-1860e8d63d7a" (UID: "4f9b70cb-bfbf-40cd-beab-1860e8d63d7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:16.935062 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.934977    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f9b70cb-bfbf-40cd-beab-1860e8d63d7a-kube-api-access-kqt5x" pod "4f9b70cb-bfbf-40cd-beab-1860e8d63d7a" (UID: "4f9b70cb-bfbf-40cd-beab-1860e8d63d7a"). InnerVolumeSpecName "kube-api-access-kqt5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:16.935406 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.935383    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acf0442b-7e18-44ad-b993-65ca686d6013-maas-api-xks-ca" pod "acf0442b-7e18-44ad-b993-65ca686d6013" (UID: "acf0442b-7e18-44ad-b993-65ca686d6013"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:16.935406 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:16.935393    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acf0442b-7e18-44ad-b993-65ca686d6013-kube-api-access-6rkfz" pod "acf0442b-7e18-44ad-b993-65ca686d6013" (UID: "acf0442b-7e18-44ad-b993-65ca686d6013"). InnerVolumeSpecName "kube-api-access-6rkfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:17.035495 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:17.035463    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rkfz\" (UniqueName: \"kubernetes.io/projected/acf0442b-7e18-44ad-b993-65ca686d6013-kube-api-access-6rkfz\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:17.035495 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:17.035493    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqt5x\" (UniqueName: \"kubernetes.io/projected/4f9b70cb-bfbf-40cd-beab-1860e8d63d7a-kube-api-access-kqt5x\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:17.035495 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:17.035503    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f9b70cb-bfbf-40cd-beab-1860e8d63d7a-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:17.035713 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:17.035512    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acf0442b-7e18-44ad-b993-65ca686d6013-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:17.169624 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:14:17.169525    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:14:17.631249 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:17.631222    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4c98b-b52m5"
Oct 30 22:14:17.631249 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:17.631239    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1dc4c98b-b52m5" event={"ID":"4f9b70cb-bfbf-40cd-beab-1860e8d63d7a","Type":"ContainerDied","Data":"4f180f659e0bc5209afdb45fd8a1d941e84d4cd59306fffc06987e4e07e685fd"}
Oct 30 22:14:17.631782 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:17.631322    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f180f659e0bc5209afdb45fd8a1d941e84d4cd59306fffc06987e4e07e685fd"
Oct 30 22:14:17.632852 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:17.632821    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ea03cd-zzd66"
Oct 30 22:14:17.632990 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:17.632860    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4ea03cd-zzd66" event={"ID":"acf0442b-7e18-44ad-b993-65ca686d6013","Type":"ContainerDied","Data":"1eee06f8927ba14d8f539159984d0552479c4657a5308a45699572314a24947e"}
Oct 30 22:14:17.632990 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:17.632901    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1eee06f8927ba14d8f539159984d0552479c4657a5308a45699572314a24947e"
Oct 30 22:14:17.709058 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:17.709029    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" path="/var/lib/kubelet/pods/699a4ca0-a268-4109-a8f8-dde6bf132689/volumes"
Oct 30 22:14:17.709498 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:17.709483    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" path="/var/lib/kubelet/pods/b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf/volumes"
Oct 30 22:14:29.429152 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:29.429108    5087 scope.go:122] "RemoveContainer" containerID="5cc2e7a3668b0e98a898b505c66c90fb34ecb782464f4c0a1ca8e80175d02e67"
Oct 30 22:14:29.439503 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:29.439482    5087 scope.go:122] "RemoveContainer" containerID="0b466c23455d3ba73bcb706f91b4ad6a5d0cb4c8f745256b45d4967a40e3e64f"
Oct 30 22:14:29.448257 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:29.448240    5087 scope.go:122] "RemoveContainer" containerID="39f9e758e796c2c0a47624913a6194d955e4b7be579bb40a81695d84e8cc792d"
Oct 30 22:14:29.457412 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:29.457395    5087 scope.go:122] "RemoveContainer" containerID="91fb4fa02180032e4ad01b5f8b7192109bee67f7d3b27bbcb0fa150799cd835f"
Oct 30 22:14:33.096964 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:33.096934    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c79cb5-464sn"]
Oct 30 22:14:33.097539 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:33.097518    5087 cpu_manager.go:414] "containerMap: removing container" podUID="4f9b70cb-bfbf-40cd-beab-1860e8d63d7a" containerName="revoke-keys"
Oct 30 22:14:33.097539 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:33.097539    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f9b70cb-bfbf-40cd-beab-1860e8d63d7a" containerName="revoke-keys"
Oct 30 22:14:33.097663 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:33.097553    5087 cpu_manager.go:414] "containerMap: removing container" podUID="acf0442b-7e18-44ad-b993-65ca686d6013" containerName="revoke-keys"
Oct 30 22:14:33.097663 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:33.097558    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acf0442b-7e18-44ad-b993-65ca686d6013" containerName="revoke-keys"
Oct 30 22:14:33.097663 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:33.097567    5087 cpu_manager.go:414] "containerMap: removing container" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerName="main"
Oct 30 22:14:33.097663 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:33.097572    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerName="main"
Oct 30 22:14:33.097663 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:33.097603    5087 cpu_manager.go:414] "containerMap: removing container" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerName="main"
Oct 30 22:14:33.097663 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:33.097608    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerName="main"
Oct 30 22:14:33.097849 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:33.097685    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7aea1c0-4e06-4f42-8a0a-ef82ae5e44bf" containerName="main"
Oct 30 22:14:33.097849 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:33.097695    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f9b70cb-bfbf-40cd-beab-1860e8d63d7a" containerName="revoke-keys"
Oct 30 22:14:33.097849 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:33.097701    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="acf0442b-7e18-44ad-b993-65ca686d6013" containerName="revoke-keys"
Oct 30 22:14:33.097849 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:33.097707    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="699a4ca0-a268-4109-a8f8-dde6bf132689" containerName="main"
Oct 30 22:14:33.102819 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:33.102800    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c79cb5-464sn"
Oct 30 22:14:33.156527 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:33.156496    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:14:33.205606 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:33.205575    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:14:33.209164 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:33.209137    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c79cb5-464sn"]
Oct 30 22:14:33.234184 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:33.234157    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjrqg\" (UniqueName: \"kubernetes.io/projected/7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f-kube-api-access-sjrqg\") pod \"maas-api-revoke-keys-e2e-worker-w2-c79cb5-464sn\" (UID: \"7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c79cb5-464sn"
Oct 30 22:14:33.234366 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:33.234232    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c79cb5-464sn\" (UID: \"7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c79cb5-464sn"
Oct 30 22:14:33.234465 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:33.234439    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c79cb5-464sn\" (UID: \"7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c79cb5-464sn"
Oct 30 22:14:33.335614 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:33.335583    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c79cb5-464sn\" (UID: \"7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c79cb5-464sn"
Oct 30 22:14:33.335834 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:33.335661    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjrqg\" (UniqueName: \"kubernetes.io/projected/7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f-kube-api-access-sjrqg\") pod \"maas-api-revoke-keys-e2e-worker-w2-c79cb5-464sn\" (UID: \"7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c79cb5-464sn"
Oct 30 22:14:33.335834 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:33.335723    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c79cb5-464sn\" (UID: \"7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c79cb5-464sn"
Oct 30 22:14:33.336317 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:33.336290    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c79cb5-464sn\" (UID: \"7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c79cb5-464sn"
Oct 30 22:14:33.338385 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:33.338331    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c79cb5-464sn\" (UID: \"7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c79cb5-464sn"
Oct 30 22:14:33.392723 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:33.392654    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjrqg\" (UniqueName: \"kubernetes.io/projected/7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f-kube-api-access-sjrqg\") pod \"maas-api-revoke-keys-e2e-worker-w2-c79cb5-464sn\" (UID: \"7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c79cb5-464sn"
Oct 30 22:14:33.416517 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:33.416486    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c79cb5-464sn"
Oct 30 22:14:33.798812 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:33.798787    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c79cb5-464sn"]
Oct 30 22:14:33.800487 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:14:33.800461    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c6aa6a7_616a_49ba_b9b0_edf1d4f8527f.slice/crio-74621c3e8fcdf2dcfe14010f9c615c065e9b22fa0ac456086a77f52c21599760 WatchSource:0}: Error finding container 74621c3e8fcdf2dcfe14010f9c615c065e9b22fa0ac456086a77f52c21599760: Status 404 returned error can't find the container with id 74621c3e8fcdf2dcfe14010f9c615c065e9b22fa0ac456086a77f52c21599760
Oct 30 22:14:34.760959 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:34.760919    5087 generic.go:356] "Generic (PLEG): container finished" podID="7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f" containerID="702f157e34bc0576a404c4965ccaa2e506fe105c1ea877fe2b1c79f1914285e4" exitCode=0
Oct 30 22:14:34.761383 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:34.760996    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c79cb5-464sn" event={"ID":"7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f","Type":"ContainerDied","Data":"702f157e34bc0576a404c4965ccaa2e506fe105c1ea877fe2b1c79f1914285e4"}
Oct 30 22:14:34.761383 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:34.761042    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c79cb5-464sn" event={"ID":"7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f","Type":"ContainerStarted","Data":"74621c3e8fcdf2dcfe14010f9c615c065e9b22fa0ac456086a77f52c21599760"}
Oct 30 22:14:35.993161 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:35.993129    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c79cb5-464sn"
Oct 30 22:14:36.122279 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:36.122196    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f-maas-api-xks-ca\") pod \"7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f\" (UID: \"7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f\") "
Oct 30 22:14:36.122475 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:36.122290    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f-kube-api-access-sjrqg\" (UniqueName: \"kubernetes.io/projected/7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f-kube-api-access-sjrqg\") pod \"7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f\" (UID: \"7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f\") "
Oct 30 22:14:36.122475 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:36.122320    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f-maas-api-openshift-ca\") pod \"7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f\" (UID: \"7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f\") "
Oct 30 22:14:36.123002 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:36.122974    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f-maas-api-openshift-ca" pod "7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f" (UID: "7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:36.124899 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:36.124875    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f-maas-api-xks-ca" pod "7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f" (UID: "7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:36.125303 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:36.125280    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f-kube-api-access-sjrqg" pod "7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f" (UID: "7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f"). InnerVolumeSpecName "kube-api-access-sjrqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:36.223525 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:36.223491    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjrqg\" (UniqueName: \"kubernetes.io/projected/7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f-kube-api-access-sjrqg\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:36.223525 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:36.223520    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:36.223746 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:36.223536    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:36.793439 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:36.793407    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c79cb5-464sn"
Oct 30 22:14:36.793631 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:36.793405    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c79cb5-464sn" event={"ID":"7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f","Type":"ContainerDied","Data":"74621c3e8fcdf2dcfe14010f9c615c065e9b22fa0ac456086a77f52c21599760"}
Oct 30 22:14:36.793631 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:36.793503    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74621c3e8fcdf2dcfe14010f9c615c065e9b22fa0ac456086a77f52c21599760"
Oct 30 22:14:37.070521 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.070446    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"]
Oct 30 22:14:37.071002 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.070987    5087 cpu_manager.go:414] "containerMap: removing container" podUID="7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f" containerName="revoke-keys"
Oct 30 22:14:37.071070 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.071003    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f" containerName="revoke-keys"
Oct 30 22:14:37.071110 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.071103    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f" containerName="revoke-keys"
Oct 30 22:14:37.074554 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.074536    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"
Oct 30 22:14:37.131250 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.131202    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvfcx\" (UniqueName: \"kubernetes.io/projected/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-kube-api-access-vvfcx\") pod \"test-model-9da607-kserve-5cdb7cd6d9-l8bsw\" (UID: \"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357\") " pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"
Oct 30 22:14:37.131587 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.131294    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-dshm\") pod \"test-model-9da607-kserve-5cdb7cd6d9-l8bsw\" (UID: \"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357\") " pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"
Oct 30 22:14:37.131587 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.131422    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-tmp-dir\") pod \"test-model-9da607-kserve-5cdb7cd6d9-l8bsw\" (UID: \"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357\") " pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"
Oct 30 22:14:37.131587 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.131458    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9da607\"/\"openshift-service-ca.crt\""
Oct 30 22:14:37.131587 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.131484    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-tls-certs\") pod \"test-model-9da607-kserve-5cdb7cd6d9-l8bsw\" (UID: \"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357\") " pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"
Oct 30 22:14:37.131804 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.131605    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-home\") pod \"test-model-9da607-kserve-5cdb7cd6d9-l8bsw\" (UID: \"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357\") " pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"
Oct 30 22:14:37.131804 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.131640    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-model-cache\") pod \"test-model-9da607-kserve-5cdb7cd6d9-l8bsw\" (UID: \"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357\") " pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"
Oct 30 22:14:37.134911 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.134886    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"]
Oct 30 22:14:37.180234 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.180203    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-9da607\"/\"kube-root-ca.crt\""
Oct 30 22:14:37.180446 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.180237    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9da607\"/\"default-dockercfg-bxr7m\""
Oct 30 22:14:37.180446 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.180249    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-9da607\"/\"test-model-9da607-kserve-self-signed-certs\""
Oct 30 22:14:37.233146 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.233110    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-tmp-dir\") pod \"test-model-9da607-kserve-5cdb7cd6d9-l8bsw\" (UID: \"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357\") " pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"
Oct 30 22:14:37.233491 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.233459    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-tls-certs\") pod \"test-model-9da607-kserve-5cdb7cd6d9-l8bsw\" (UID: \"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357\") " pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"
Oct 30 22:14:37.233649 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.233521    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-home\") pod \"test-model-9da607-kserve-5cdb7cd6d9-l8bsw\" (UID: \"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357\") " pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"
Oct 30 22:14:37.233649 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.233561    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-model-cache\") pod \"test-model-9da607-kserve-5cdb7cd6d9-l8bsw\" (UID: \"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357\") " pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"
Oct 30 22:14:37.234362 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.233680    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvfcx\" (UniqueName: \"kubernetes.io/projected/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-kube-api-access-vvfcx\") pod \"test-model-9da607-kserve-5cdb7cd6d9-l8bsw\" (UID: \"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357\") " pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"
Oct 30 22:14:37.234362 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.233745    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-dshm\") pod \"test-model-9da607-kserve-5cdb7cd6d9-l8bsw\" (UID: \"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357\") " pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"
Oct 30 22:14:37.234362 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.233878    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-tmp-dir\") pod \"test-model-9da607-kserve-5cdb7cd6d9-l8bsw\" (UID: \"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357\") " pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"
Oct 30 22:14:37.234362 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.234229    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-home\") pod \"test-model-9da607-kserve-5cdb7cd6d9-l8bsw\" (UID: \"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357\") " pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"
Oct 30 22:14:37.235924 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.235901    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-model-cache\") pod \"test-model-9da607-kserve-5cdb7cd6d9-l8bsw\" (UID: \"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357\") " pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"
Oct 30 22:14:37.243249 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.243165    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-dshm\") pod \"test-model-9da607-kserve-5cdb7cd6d9-l8bsw\" (UID: \"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357\") " pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"
Oct 30 22:14:37.244852 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.244831    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-tls-certs\") pod \"test-model-9da607-kserve-5cdb7cd6d9-l8bsw\" (UID: \"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357\") " pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"
Oct 30 22:14:37.293782 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.293745    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvfcx\" (UniqueName: \"kubernetes.io/projected/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-kube-api-access-vvfcx\") pod \"test-model-9da607-kserve-5cdb7cd6d9-l8bsw\" (UID: \"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357\") " pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"
Oct 30 22:14:37.387434 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.387329    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"
Oct 30 22:14:37.573372 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.573118    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"]
Oct 30 22:14:37.599256 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.599201    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"]
Oct 30 22:14:37.774392 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.774366    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"]
Oct 30 22:14:37.775488 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:14:37.775461    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod368c14ec_f6ad_4c3e_bbaa_ca88e9cc1357.slice/crio-cbb380428e6b884909353038dba57019a40d37a3c3c95bf122b0f030e747bc82 WatchSource:0}: Error finding container cbb380428e6b884909353038dba57019a40d37a3c3c95bf122b0f030e747bc82: Status 404 returned error can't find the container with id cbb380428e6b884909353038dba57019a40d37a3c3c95bf122b0f030e747bc82
Oct 30 22:14:37.805103 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:37.805073    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw" event={"ID":"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357","Type":"ContainerStarted","Data":"cbb380428e6b884909353038dba57019a40d37a3c3c95bf122b0f030e747bc82"}
Oct 30 22:14:38.172666 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:38.172576    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c030d4a4-k4mhh"]
Oct 30 22:14:38.187544 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:38.187517    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c030d4a4-k4mhh"]
Oct 30 22:14:38.813210 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:38.813172    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw" event={"ID":"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357","Type":"ContainerStarted","Data":"641fdd7935343a26f8c0471147532f0a875660cc7321b4446a65ec864eacb05a"}
Oct 30 22:14:38.934911 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:38.934857    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw" podStartSLOduration=1.934841385 podStartE2EDuration="1.934841385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:38.932918997 +0000 UTC m=+2473.407422976" watchObservedRunningTime="2026-10-30 22:14:38.934841385 +0000 UTC m=+2473.409345368"
Oct 30 22:14:39.387715 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:39.387678    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"
Oct 30 22:14:39.710807 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:39.710731    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e6825e7-85c9-4cba-8f6e-e52669c32616" path="/var/lib/kubelet/pods/9e6825e7-85c9-4cba-8f6e-e52669c32616/volumes"
Oct 30 22:14:47.169033 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:14:47.169000    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:14:47.387793 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:47.387764    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"
Oct 30 22:14:47.400494 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:47.400464    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"
Oct 30 22:14:47.884520 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:47.884490    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"
Oct 30 22:14:52.632751 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:52.632709    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main" containerID="cri-o://cd2e69f347fc6ebeea80caac1a39009db474d9edcc0145d5bcd9da1f22c5db5a" gracePeriod=45
Oct 30 22:14:52.633452 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:52.633335    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main" containerID="cri-o://5c769f0a046092765d98c06e4d5e669273856ac9f991fd7b9c829bd78174769f" gracePeriod=45
Oct 30 22:14:52.919876 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:52.919847    5087 generic.go:356] "Generic (PLEG): container finished" podID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerID="5c769f0a046092765d98c06e4d5e669273856ac9f991fd7b9c829bd78174769f" exitCode=0
Oct 30 22:14:52.920063 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:52.919939    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" event={"ID":"06f6da2e-4915-493e-b4fe-f06c2b8612e5","Type":"ContainerDied","Data":"5c769f0a046092765d98c06e4d5e669273856ac9f991fd7b9c829bd78174769f"}
Oct 30 22:14:52.921843 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:52.921815    5087 generic.go:356] "Generic (PLEG): container finished" podID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerID="cd2e69f347fc6ebeea80caac1a39009db474d9edcc0145d5bcd9da1f22c5db5a" exitCode=0
Oct 30 22:14:52.921965 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:52.921887    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" event={"ID":"07695dfe-10c8-44af-89c5-cbb19eb57ddf","Type":"ContainerDied","Data":"cd2e69f347fc6ebeea80caac1a39009db474d9edcc0145d5bcd9da1f22c5db5a"}
Oct 30 22:14:52.921965 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:52.921921    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg" event={"ID":"07695dfe-10c8-44af-89c5-cbb19eb57ddf","Type":"ContainerDied","Data":"ce389a910115f20b7e5fb483d603aba6c0fae19a8a2523dbb582b0afcb316ff1"}
Oct 30 22:14:52.921965 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:52.921935    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce389a910115f20b7e5fb483d603aba6c0fae19a8a2523dbb582b0afcb316ff1"
Oct 30 22:14:52.944395 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:52.944370    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 22:14:52.948341 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:52.948324    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 22:14:53.058762 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.058726    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07695dfe-10c8-44af-89c5-cbb19eb57ddf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/07695dfe-10c8-44af-89c5-cbb19eb57ddf-dshm\") pod \"07695dfe-10c8-44af-89c5-cbb19eb57ddf\" (UID: \"07695dfe-10c8-44af-89c5-cbb19eb57ddf\") "
Oct 30 22:14:53.058987 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.058807    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07695dfe-10c8-44af-89c5-cbb19eb57ddf-home\" (UniqueName: \"kubernetes.io/empty-dir/07695dfe-10c8-44af-89c5-cbb19eb57ddf-home\") pod \"07695dfe-10c8-44af-89c5-cbb19eb57ddf\" (UID: \"07695dfe-10c8-44af-89c5-cbb19eb57ddf\") "
Oct 30 22:14:53.058987 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.058851    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07695dfe-10c8-44af-89c5-cbb19eb57ddf-tls-certs\" (UniqueName: \"kubernetes.io/secret/07695dfe-10c8-44af-89c5-cbb19eb57ddf-tls-certs\") pod \"07695dfe-10c8-44af-89c5-cbb19eb57ddf\" (UID: \"07695dfe-10c8-44af-89c5-cbb19eb57ddf\") "
Oct 30 22:14:53.058987 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.058937    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f6da2e-4915-493e-b4fe-f06c2b8612e5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/06f6da2e-4915-493e-b4fe-f06c2b8612e5-dshm\") pod \"06f6da2e-4915-493e-b4fe-f06c2b8612e5\" (UID: \"06f6da2e-4915-493e-b4fe-f06c2b8612e5\") "
Oct 30 22:14:53.058987 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.058977    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06f6da2e-4915-493e-b4fe-f06c2b8612e5-kube-api-access-5r76w\" (UniqueName: \"kubernetes.io/projected/06f6da2e-4915-493e-b4fe-f06c2b8612e5-kube-api-access-5r76w\") pod \"06f6da2e-4915-493e-b4fe-f06c2b8612e5\" (UID: \"06f6da2e-4915-493e-b4fe-f06c2b8612e5\") "
Oct 30 22:14:53.059211 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.059007    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07695dfe-10c8-44af-89c5-cbb19eb57ddf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07695dfe-10c8-44af-89c5-cbb19eb57ddf-model-cache\") pod \"07695dfe-10c8-44af-89c5-cbb19eb57ddf\" (UID: \"07695dfe-10c8-44af-89c5-cbb19eb57ddf\") "
Oct 30 22:14:53.059211 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.059061    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07695dfe-10c8-44af-89c5-cbb19eb57ddf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07695dfe-10c8-44af-89c5-cbb19eb57ddf-tmp-dir\") pod \"07695dfe-10c8-44af-89c5-cbb19eb57ddf\" (UID: \"07695dfe-10c8-44af-89c5-cbb19eb57ddf\") "
Oct 30 22:14:53.059211 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.059091    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f6da2e-4915-493e-b4fe-f06c2b8612e5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f6da2e-4915-493e-b4fe-f06c2b8612e5-tmp-dir\") pod \"06f6da2e-4915-493e-b4fe-f06c2b8612e5\" (UID: \"06f6da2e-4915-493e-b4fe-f06c2b8612e5\") "
Oct 30 22:14:53.059211 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.059180    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f6da2e-4915-493e-b4fe-f06c2b8612e5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06f6da2e-4915-493e-b4fe-f06c2b8612e5-model-cache\") pod \"06f6da2e-4915-493e-b4fe-f06c2b8612e5\" (UID: \"06f6da2e-4915-493e-b4fe-f06c2b8612e5\") "
Oct 30 22:14:53.059453 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.059229    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06f6da2e-4915-493e-b4fe-f06c2b8612e5-tls-certs\" (UniqueName: \"kubernetes.io/secret/06f6da2e-4915-493e-b4fe-f06c2b8612e5-tls-certs\") pod \"06f6da2e-4915-493e-b4fe-f06c2b8612e5\" (UID: \"06f6da2e-4915-493e-b4fe-f06c2b8612e5\") "
Oct 30 22:14:53.059453 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.059282    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06f6da2e-4915-493e-b4fe-f06c2b8612e5-home\" (UniqueName: \"kubernetes.io/empty-dir/06f6da2e-4915-493e-b4fe-f06c2b8612e5-home\") pod \"06f6da2e-4915-493e-b4fe-f06c2b8612e5\" (UID: \"06f6da2e-4915-493e-b4fe-f06c2b8612e5\") "
Oct 30 22:14:53.059453 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.059315    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07695dfe-10c8-44af-89c5-cbb19eb57ddf-kube-api-access-pwdkd\" (UniqueName: \"kubernetes.io/projected/07695dfe-10c8-44af-89c5-cbb19eb57ddf-kube-api-access-pwdkd\") pod \"07695dfe-10c8-44af-89c5-cbb19eb57ddf\" (UID: \"07695dfe-10c8-44af-89c5-cbb19eb57ddf\") "
Oct 30 22:14:53.060047 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.060016    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f6da2e-4915-493e-b4fe-f06c2b8612e5-home" pod "06f6da2e-4915-493e-b4fe-f06c2b8612e5" (UID: "06f6da2e-4915-493e-b4fe-f06c2b8612e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:53.060247 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.060225    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07695dfe-10c8-44af-89c5-cbb19eb57ddf-tmp-dir" pod "07695dfe-10c8-44af-89c5-cbb19eb57ddf" (UID: "07695dfe-10c8-44af-89c5-cbb19eb57ddf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:53.061235 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.060842    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f6da2e-4915-493e-b4fe-f06c2b8612e5-tmp-dir" pod "06f6da2e-4915-493e-b4fe-f06c2b8612e5" (UID: "06f6da2e-4915-493e-b4fe-f06c2b8612e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:53.061235 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.060904    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07695dfe-10c8-44af-89c5-cbb19eb57ddf-home" pod "07695dfe-10c8-44af-89c5-cbb19eb57ddf" (UID: "07695dfe-10c8-44af-89c5-cbb19eb57ddf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:53.062195 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.062021    5087 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07695dfe-10c8-44af-89c5-cbb19eb57ddf-tmp-dir\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:53.062195 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.062049    5087 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06f6da2e-4915-493e-b4fe-f06c2b8612e5-tmp-dir\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:53.062195 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.062062    5087 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06f6da2e-4915-493e-b4fe-f06c2b8612e5-home\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:53.062195 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.062073    5087 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07695dfe-10c8-44af-89c5-cbb19eb57ddf-home\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:53.062195 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.062125    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f6da2e-4915-493e-b4fe-f06c2b8612e5-model-cache" pod "06f6da2e-4915-493e-b4fe-f06c2b8612e5" (UID: "06f6da2e-4915-493e-b4fe-f06c2b8612e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:53.062195 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.062169    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07695dfe-10c8-44af-89c5-cbb19eb57ddf-model-cache" pod "07695dfe-10c8-44af-89c5-cbb19eb57ddf" (UID: "07695dfe-10c8-44af-89c5-cbb19eb57ddf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:53.064711 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.064662    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07695dfe-10c8-44af-89c5-cbb19eb57ddf-dshm" pod "07695dfe-10c8-44af-89c5-cbb19eb57ddf" (UID: "07695dfe-10c8-44af-89c5-cbb19eb57ddf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:53.066006 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.065965    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07695dfe-10c8-44af-89c5-cbb19eb57ddf-tls-certs" pod "07695dfe-10c8-44af-89c5-cbb19eb57ddf" (UID: "07695dfe-10c8-44af-89c5-cbb19eb57ddf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:53.066121 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.066098    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06f6da2e-4915-493e-b4fe-f06c2b8612e5-tls-certs" pod "06f6da2e-4915-493e-b4fe-f06c2b8612e5" (UID: "06f6da2e-4915-493e-b4fe-f06c2b8612e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:53.067451 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.067413    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f6da2e-4915-493e-b4fe-f06c2b8612e5-kube-api-access-5r76w" pod "06f6da2e-4915-493e-b4fe-f06c2b8612e5" (UID: "06f6da2e-4915-493e-b4fe-f06c2b8612e5"). InnerVolumeSpecName "kube-api-access-5r76w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:53.067560 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.067492    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06f6da2e-4915-493e-b4fe-f06c2b8612e5-dshm" pod "06f6da2e-4915-493e-b4fe-f06c2b8612e5" (UID: "06f6da2e-4915-493e-b4fe-f06c2b8612e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:14:53.067741 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.067719    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07695dfe-10c8-44af-89c5-cbb19eb57ddf-kube-api-access-pwdkd" pod "07695dfe-10c8-44af-89c5-cbb19eb57ddf" (UID: "07695dfe-10c8-44af-89c5-cbb19eb57ddf"). InnerVolumeSpecName "kube-api-access-pwdkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:53.164061 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.163973    5087 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07695dfe-10c8-44af-89c5-cbb19eb57ddf-model-cache\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:53.164061 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.164010    5087 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06f6da2e-4915-493e-b4fe-f06c2b8612e5-model-cache\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:53.164061 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.164025    5087 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06f6da2e-4915-493e-b4fe-f06c2b8612e5-tls-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:53.164061 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.164043    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwdkd\" (UniqueName: \"kubernetes.io/projected/07695dfe-10c8-44af-89c5-cbb19eb57ddf-kube-api-access-pwdkd\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:53.164061 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.164062    5087 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07695dfe-10c8-44af-89c5-cbb19eb57ddf-dshm\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:53.164426 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.164075    5087 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07695dfe-10c8-44af-89c5-cbb19eb57ddf-tls-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:53.164426 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.164087    5087 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06f6da2e-4915-493e-b4fe-f06c2b8612e5-dshm\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:53.164426 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.164100    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r76w\" (UniqueName: \"kubernetes.io/projected/06f6da2e-4915-493e-b4fe-f06c2b8612e5-kube-api-access-5r76w\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:14:53.930086 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.930012    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"
Oct 30 22:14:53.930086 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.930049    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9" event={"ID":"06f6da2e-4915-493e-b4fe-f06c2b8612e5","Type":"ContainerDied","Data":"e6ebd25008f7d790d1e3ac1d08925640d6c1da126076a1731ecbef60e5046c41"}
Oct 30 22:14:53.930580 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.930104    5087 scope.go:122] "RemoveContainer" containerID="5c769f0a046092765d98c06e4d5e669273856ac9f991fd7b9c829bd78174769f"
Oct 30 22:14:53.930580 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:53.930142    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"
Oct 30 22:14:54.148848 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:54.148817    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"]
Oct 30 22:14:54.198715 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:54.198640    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-c79cb5/distinct-w2-c79cb5-kserve-59bfccfbc6-4bhwg"]
Oct 30 22:14:54.265589 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:54.265536    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"]
Oct 30 22:14:54.269361 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:54.269324    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-c79cb5/simulator-w2-c79cb5-kserve-fbbb584fb-jscm9"]
Oct 30 22:14:55.709251 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:55.709215    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" path="/var/lib/kubelet/pods/06f6da2e-4915-493e-b4fe-f06c2b8612e5/volumes"
Oct 30 22:14:55.709865 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:14:55.709845    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" path="/var/lib/kubelet/pods/07695dfe-10c8-44af-89c5-cbb19eb57ddf/volumes"
Oct 30 22:15:00.197629 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.197592    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l"]
Oct 30 22:15:00.198140 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.198126    5087 cpu_manager.go:414] "containerMap: removing container" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main"
Oct 30 22:15:00.198185 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.198140    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main"
Oct 30 22:15:00.198185 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.198155    5087 cpu_manager.go:414] "containerMap: removing container" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main"
Oct 30 22:15:00.198185 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.198160    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main"
Oct 30 22:15:00.198283 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.198239    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="06f6da2e-4915-493e-b4fe-f06c2b8612e5" containerName="main"
Oct 30 22:15:00.198283 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.198249    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="07695dfe-10c8-44af-89c5-cbb19eb57ddf" containerName="main"
Oct 30 22:15:00.201683 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.201665    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l"
Oct 30 22:15:00.203741 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.203722    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889975-vwsk4"]
Oct 30 22:15:00.208080 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.208063    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt"]
Oct 30 22:15:00.208201 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.208182    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889975-vwsk4"
Oct 30 22:15:00.212527 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.212508    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt"
Oct 30 22:15:00.249612 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.249582    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj"]
Oct 30 22:15:00.254234 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.254212    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj"
Oct 30 22:15:00.254666 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.254646    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:15:00.297162 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.297129    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889975-vwsk4"]
Oct 30 22:15:00.322371 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.315287    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt"]
Oct 30 22:15:00.322371 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.315389    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l"]
Oct 30 22:15:00.349613 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.349581    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b01d57d-4936-4e45-8b9e-c9ea8e7246fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l\" (UID: \"1b01d57d-4936-4e45-8b9e-c9ea8e7246fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l"
Oct 30 22:15:00.349842 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.349821    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9hkr\" (UniqueName: \"kubernetes.io/projected/dce092df-6490-433b-a5fa-02df7db0b5a8-kube-api-access-t9hkr\") pod \"maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889975-vwsk4\" (UID: \"dce092df-6490-433b-a5fa-02df7db0b5a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889975-vwsk4"
Oct 30 22:15:00.350017 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.349973    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm4gh\" (UniqueName: \"kubernetes.io/projected/830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca-kube-api-access-rm4gh\") pod \"maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt\" (UID: \"830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt"
Oct 30 22:15:00.350472 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.350450    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phgrb\" (UniqueName: \"kubernetes.io/projected/1b01d57d-4936-4e45-8b9e-c9ea8e7246fc-kube-api-access-phgrb\") pod \"maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l\" (UID: \"1b01d57d-4936-4e45-8b9e-c9ea8e7246fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l"
Oct 30 22:15:00.350691 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.350667    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt\" (UID: \"830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt"
Oct 30 22:15:00.350771 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.350711    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dce092df-6490-433b-a5fa-02df7db0b5a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889975-vwsk4\" (UID: \"dce092df-6490-433b-a5fa-02df7db0b5a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889975-vwsk4"
Oct 30 22:15:00.350771 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.350758    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj"]
Oct 30 22:15:00.451968 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.451877    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb54j\" (UniqueName: \"kubernetes.io/projected/1b48210a-16e2-423e-a233-0d340be66ff5-kube-api-access-fb54j\") pod \"maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj\" (UID: \"1b48210a-16e2-423e-a233-0d340be66ff5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj"
Oct 30 22:15:00.452253 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.452224    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b01d57d-4936-4e45-8b9e-c9ea8e7246fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l\" (UID: \"1b01d57d-4936-4e45-8b9e-c9ea8e7246fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l"
Oct 30 22:15:00.452418 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.452268    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9hkr\" (UniqueName: \"kubernetes.io/projected/dce092df-6490-433b-a5fa-02df7db0b5a8-kube-api-access-t9hkr\") pod \"maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889975-vwsk4\" (UID: \"dce092df-6490-433b-a5fa-02df7db0b5a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889975-vwsk4"
Oct 30 22:15:00.452418 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.452294    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm4gh\" (UniqueName: \"kubernetes.io/projected/830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca-kube-api-access-rm4gh\") pod \"maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt\" (UID: \"830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt"
Oct 30 22:15:00.452418 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.452360    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b48210a-16e2-423e-a233-0d340be66ff5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj\" (UID: \"1b48210a-16e2-423e-a233-0d340be66ff5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj"
Oct 30 22:15:00.452595 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.452503    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phgrb\" (UniqueName: \"kubernetes.io/projected/1b01d57d-4936-4e45-8b9e-c9ea8e7246fc-kube-api-access-phgrb\") pod \"maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l\" (UID: \"1b01d57d-4936-4e45-8b9e-c9ea8e7246fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l"
Oct 30 22:15:00.452595 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.452558    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt\" (UID: \"830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt"
Oct 30 22:15:00.452705 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.452591    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dce092df-6490-433b-a5fa-02df7db0b5a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889975-vwsk4\" (UID: \"dce092df-6490-433b-a5fa-02df7db0b5a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889975-vwsk4"
Oct 30 22:15:00.454039 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.453982    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt\" (UID: \"830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt"
Oct 30 22:15:00.454039 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.454032    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b01d57d-4936-4e45-8b9e-c9ea8e7246fc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l\" (UID: \"1b01d57d-4936-4e45-8b9e-c9ea8e7246fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l"
Oct 30 22:15:00.454401 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.454378    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dce092df-6490-433b-a5fa-02df7db0b5a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889975-vwsk4\" (UID: \"dce092df-6490-433b-a5fa-02df7db0b5a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889975-vwsk4"
Oct 30 22:15:00.544368 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.543657    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9hkr\" (UniqueName: \"kubernetes.io/projected/dce092df-6490-433b-a5fa-02df7db0b5a8-kube-api-access-t9hkr\") pod \"maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889975-vwsk4\" (UID: \"dce092df-6490-433b-a5fa-02df7db0b5a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889975-vwsk4"
Oct 30 22:15:00.544368 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.543678    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phgrb\" (UniqueName: \"kubernetes.io/projected/1b01d57d-4936-4e45-8b9e-c9ea8e7246fc-kube-api-access-phgrb\") pod \"maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l\" (UID: \"1b01d57d-4936-4e45-8b9e-c9ea8e7246fc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l"
Oct 30 22:15:00.551372 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.550156    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm4gh\" (UniqueName: \"kubernetes.io/projected/830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca-kube-api-access-rm4gh\") pod \"maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt\" (UID: \"830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt"
Oct 30 22:15:00.564327 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.563027    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb54j\" (UniqueName: \"kubernetes.io/projected/1b48210a-16e2-423e-a233-0d340be66ff5-kube-api-access-fb54j\") pod \"maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj\" (UID: \"1b48210a-16e2-423e-a233-0d340be66ff5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj"
Oct 30 22:15:00.564327 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.563131    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b48210a-16e2-423e-a233-0d340be66ff5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj\" (UID: \"1b48210a-16e2-423e-a233-0d340be66ff5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj"
Oct 30 22:15:00.564327 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.563995    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b48210a-16e2-423e-a233-0d340be66ff5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj\" (UID: \"1b48210a-16e2-423e-a233-0d340be66ff5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj"
Oct 30 22:15:00.597739 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.597702    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889975-vwsk4"]
Oct 30 22:15:00.598362 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.598318    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889975-vwsk4"
Oct 30 22:15:00.635592 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.635214    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb54j\" (UniqueName: \"kubernetes.io/projected/1b48210a-16e2-423e-a233-0d340be66ff5-kube-api-access-fb54j\") pod \"maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj\" (UID: \"1b48210a-16e2-423e-a233-0d340be66ff5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj"
Oct 30 22:15:00.817376 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.816946    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l"
Oct 30 22:15:00.832416 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.832389    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt"
Oct 30 22:15:00.868523 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:00.868166    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj"
Oct 30 22:15:01.032230 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:01.032195    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889975-vwsk4"]
Oct 30 22:15:01.033406 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:15:01.033379    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce092df_6490_433b_a5fa_02df7db0b5a8.slice/crio-100d517de5216debb2b07d23d3f2001eda63b142bb4cdbebd744c7485cab4417 WatchSource:0}: Error finding container 100d517de5216debb2b07d23d3f2001eda63b142bb4cdbebd744c7485cab4417: Status 404 returned error can't find the container with id 100d517de5216debb2b07d23d3f2001eda63b142bb4cdbebd744c7485cab4417
Oct 30 22:15:01.240415 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:01.240360    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l"]
Oct 30 22:15:01.243416 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:15:01.243339    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b01d57d_4936_4e45_8b9e_c9ea8e7246fc.slice/crio-79c61e2ad5c4ba433d0a86f9032761114c3ee533871edc5ad4402d0baf93b3a7 WatchSource:0}: Error finding container 79c61e2ad5c4ba433d0a86f9032761114c3ee533871edc5ad4402d0baf93b3a7: Status 404 returned error can't find the container with id 79c61e2ad5c4ba433d0a86f9032761114c3ee533871edc5ad4402d0baf93b3a7
Oct 30 22:15:01.256859 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:01.256835    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt"]
Oct 30 22:15:01.257427 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:15:01.257399    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod830b2eaa_a3e3_469d_ad75_96f4d2a3a2ca.slice/crio-95d66a5d679969d667a8070fefdedf27f149e2018cc934330cff31c1f3cd935c WatchSource:0}: Error finding container 95d66a5d679969d667a8070fefdedf27f149e2018cc934330cff31c1f3cd935c: Status 404 returned error can't find the container with id 95d66a5d679969d667a8070fefdedf27f149e2018cc934330cff31c1f3cd935c
Oct 30 22:15:01.384396 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:01.379925    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj"]
Oct 30 22:15:01.461580 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:01.461532    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-557708-72cvv"]
Oct 30 22:15:01.467980 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:01.467949    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-557708-72cvv"]
Oct 30 22:15:01.709610 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:01.709579    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17f4cbc0-822c-4137-8a9a-addf48bf49e4" path="/var/lib/kubelet/pods/17f4cbc0-822c-4137-8a9a-addf48bf49e4/volumes"
Oct 30 22:15:01.993299 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:01.993263    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l" event={"ID":"1b01d57d-4936-4e45-8b9e-c9ea8e7246fc","Type":"ContainerStarted","Data":"4c42d4488b743b312ba352a4222dcf88347b52c3f853921277ae3a5a37931fc9"}
Oct 30 22:15:01.993520 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:01.993326    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l" event={"ID":"1b01d57d-4936-4e45-8b9e-c9ea8e7246fc","Type":"ContainerStarted","Data":"79c61e2ad5c4ba433d0a86f9032761114c3ee533871edc5ad4402d0baf93b3a7"}
Oct 30 22:15:01.994974 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:01.994949    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt" event={"ID":"830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca","Type":"ContainerStarted","Data":"241676cf7c59b17e85b6af34d069bc90a05c7f18642c59798cbd771d6611f9c9"}
Oct 30 22:15:01.995110 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:01.994990    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt" event={"ID":"830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca","Type":"ContainerStarted","Data":"95d66a5d679969d667a8070fefdedf27f149e2018cc934330cff31c1f3cd935c"}
Oct 30 22:15:01.996533 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:01.996508    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889975-vwsk4" event={"ID":"dce092df-6490-433b-a5fa-02df7db0b5a8","Type":"ContainerStarted","Data":"45368b8bb55e647d20e83162d52a5def7ca6cc4b5e2484bf323e4e65a560e980"}
Oct 30 22:15:01.996634 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:01.996556    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889975-vwsk4" event={"ID":"dce092df-6490-433b-a5fa-02df7db0b5a8","Type":"ContainerStarted","Data":"100d517de5216debb2b07d23d3f2001eda63b142bb4cdbebd744c7485cab4417"}
Oct 30 22:15:01.996634 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:01.996552    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889975-vwsk4" podUID="dce092df-6490-433b-a5fa-02df7db0b5a8" containerName="cleanup" containerID="cri-o://45368b8bb55e647d20e83162d52a5def7ca6cc4b5e2484bf323e4e65a560e980" gracePeriod=30
Oct 30 22:15:01.997971 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:01.997936    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj" event={"ID":"1b48210a-16e2-423e-a233-0d340be66ff5","Type":"ContainerStarted","Data":"0846efd57e3c41d47dd615e6b369723e06419739c448e824660d0a1ea74a8ef3"}
Oct 30 22:15:01.998068 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:01.997986    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj" event={"ID":"1b48210a-16e2-423e-a233-0d340be66ff5","Type":"ContainerStarted","Data":"b84437fc98c6e3e25417fc625ee0b80a104a103a7980fe327bb3b1fff6f35a81"}
Oct 30 22:15:02.144164 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:02.144102    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l" podStartSLOduration=2.144084176 podStartE2EDuration="2.144084176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.141047674 +0000 UTC m=+2496.615551654" watchObservedRunningTime="2026-10-30 22:15:02.144084176 +0000 UTC m=+2496.618588156"
Oct 30 22:15:02.262111 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:02.262052    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889975-vwsk4" podStartSLOduration=2.262031912 podStartE2EDuration="2.262031912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.261929948 +0000 UTC m=+2496.736433928" watchObservedRunningTime="2026-10-30 22:15:02.262031912 +0000 UTC m=+2496.736535892"
Oct 30 22:15:02.305872 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:02.305843    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889975-vwsk4"
Oct 30 22:15:02.385685 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:02.385638    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj" podStartSLOduration=2.385624326 podStartE2EDuration="2.385624326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.38148528 +0000 UTC m=+2496.855989259" watchObservedRunningTime="2026-10-30 22:15:02.385624326 +0000 UTC m=+2496.860128306"
Oct 30 22:15:02.400329 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:02.400298    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dce092df-6490-433b-a5fa-02df7db0b5a8-kube-api-access-t9hkr\" (UniqueName: \"kubernetes.io/projected/dce092df-6490-433b-a5fa-02df7db0b5a8-kube-api-access-t9hkr\") pod \"dce092df-6490-433b-a5fa-02df7db0b5a8\" (UID: \"dce092df-6490-433b-a5fa-02df7db0b5a8\") "
Oct 30 22:15:02.400482 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:02.400469    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dce092df-6490-433b-a5fa-02df7db0b5a8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dce092df-6490-433b-a5fa-02df7db0b5a8-maas-api-service-ca\") pod \"dce092df-6490-433b-a5fa-02df7db0b5a8\" (UID: \"dce092df-6490-433b-a5fa-02df7db0b5a8\") "
Oct 30 22:15:02.400788 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:02.400764    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce092df-6490-433b-a5fa-02df7db0b5a8-maas-api-service-ca" pod "dce092df-6490-433b-a5fa-02df7db0b5a8" (UID: "dce092df-6490-433b-a5fa-02df7db0b5a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:02.402616 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:02.402597    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce092df-6490-433b-a5fa-02df7db0b5a8-kube-api-access-t9hkr" pod "dce092df-6490-433b-a5fa-02df7db0b5a8" (UID: "dce092df-6490-433b-a5fa-02df7db0b5a8"). InnerVolumeSpecName "kube-api-access-t9hkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:02.510539 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:02.502229    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9hkr\" (UniqueName: \"kubernetes.io/projected/dce092df-6490-433b-a5fa-02df7db0b5a8-kube-api-access-t9hkr\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:02.510539 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:02.502262    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dce092df-6490-433b-a5fa-02df7db0b5a8-maas-api-service-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:02.510539 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:02.505817    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt" podStartSLOduration=2.5057984920000003 podStartE2EDuration="2.505798492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.502553124 +0000 UTC m=+2496.977057104" watchObservedRunningTime="2026-10-30 22:15:02.505798492 +0000 UTC m=+2496.980302472"
Oct 30 22:15:02.859206 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:02.859119    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b"]
Oct 30 22:15:02.864898 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:02.864865    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-c3e2d8-t795b"]
Oct 30 22:15:03.008611 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:03.008570    5087 generic.go:356] "Generic (PLEG): container finished" podID="1b01d57d-4936-4e45-8b9e-c9ea8e7246fc" containerID="4c42d4488b743b312ba352a4222dcf88347b52c3f853921277ae3a5a37931fc9" exitCode=60
Oct 30 22:15:03.008784 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:03.008698    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l" event={"ID":"1b01d57d-4936-4e45-8b9e-c9ea8e7246fc","Type":"ContainerDied","Data":"4c42d4488b743b312ba352a4222dcf88347b52c3f853921277ae3a5a37931fc9"}
Oct 30 22:15:03.009098 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:03.009076    5087 scope.go:122] "RemoveContainer" containerID="4c42d4488b743b312ba352a4222dcf88347b52c3f853921277ae3a5a37931fc9"
Oct 30 22:15:03.011882 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:03.011853    5087 generic.go:356] "Generic (PLEG): container finished" podID="830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca" containerID="241676cf7c59b17e85b6af34d069bc90a05c7f18642c59798cbd771d6611f9c9" exitCode=60
Oct 30 22:15:03.011986 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:03.011899    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt" event={"ID":"830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca","Type":"ContainerDied","Data":"241676cf7c59b17e85b6af34d069bc90a05c7f18642c59798cbd771d6611f9c9"}
Oct 30 22:15:03.012448 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:03.012329    5087 scope.go:122] "RemoveContainer" containerID="241676cf7c59b17e85b6af34d069bc90a05c7f18642c59798cbd771d6611f9c9"
Oct 30 22:15:03.013600 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:03.013570    5087 generic.go:356] "Generic (PLEG): container finished" podID="dce092df-6490-433b-a5fa-02df7db0b5a8" containerID="45368b8bb55e647d20e83162d52a5def7ca6cc4b5e2484bf323e4e65a560e980" exitCode=6
Oct 30 22:15:03.013699 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:03.013617    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889975-vwsk4"
Oct 30 22:15:03.013699 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:03.013651    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889975-vwsk4" event={"ID":"dce092df-6490-433b-a5fa-02df7db0b5a8","Type":"ContainerDied","Data":"45368b8bb55e647d20e83162d52a5def7ca6cc4b5e2484bf323e4e65a560e980"}
Oct 30 22:15:03.013699 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:03.013690    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889975-vwsk4" event={"ID":"dce092df-6490-433b-a5fa-02df7db0b5a8","Type":"ContainerDied","Data":"100d517de5216debb2b07d23d3f2001eda63b142bb4cdbebd744c7485cab4417"}
Oct 30 22:15:03.013958 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:03.013711    5087 scope.go:122] "RemoveContainer" containerID="45368b8bb55e647d20e83162d52a5def7ca6cc4b5e2484bf323e4e65a560e980"
Oct 30 22:15:03.016094 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:03.016067    5087 generic.go:356] "Generic (PLEG): container finished" podID="1b48210a-16e2-423e-a233-0d340be66ff5" containerID="0846efd57e3c41d47dd615e6b369723e06419739c448e824660d0a1ea74a8ef3" exitCode=60
Oct 30 22:15:03.016206 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:03.016179    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj" event={"ID":"1b48210a-16e2-423e-a233-0d340be66ff5","Type":"ContainerDied","Data":"0846efd57e3c41d47dd615e6b369723e06419739c448e824660d0a1ea74a8ef3"}
Oct 30 22:15:03.016529 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:03.016512    5087 scope.go:122] "RemoveContainer" containerID="0846efd57e3c41d47dd615e6b369723e06419739c448e824660d0a1ea74a8ef3"
Oct 30 22:15:03.025667 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:03.025643    5087 scope.go:122] "RemoveContainer" containerID="45368b8bb55e647d20e83162d52a5def7ca6cc4b5e2484bf323e4e65a560e980"
Oct 30 22:15:03.025991 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:15:03.025965    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45368b8bb55e647d20e83162d52a5def7ca6cc4b5e2484bf323e4e65a560e980\": container with ID starting with 45368b8bb55e647d20e83162d52a5def7ca6cc4b5e2484bf323e4e65a560e980 not found: ID does not exist" containerID="45368b8bb55e647d20e83162d52a5def7ca6cc4b5e2484bf323e4e65a560e980"
Oct 30 22:15:03.026092 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:03.025999    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45368b8bb55e647d20e83162d52a5def7ca6cc4b5e2484bf323e4e65a560e980"} err="failed to get container status \"45368b8bb55e647d20e83162d52a5def7ca6cc4b5e2484bf323e4e65a560e980\": rpc error: code = NotFound desc = could not find container \"45368b8bb55e647d20e83162d52a5def7ca6cc4b5e2484bf323e4e65a560e980\": container with ID starting with 45368b8bb55e647d20e83162d52a5def7ca6cc4b5e2484bf323e4e65a560e980 not found: ID does not exist"
Oct 30 22:15:03.366692 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:03.366660    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889975-vwsk4"]
Oct 30 22:15:03.373064 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:03.373039    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-c3e2d8-29889975-vwsk4"]
Oct 30 22:15:03.712404 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:03.712311    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fe89e6c-771c-4cb7-a4f5-54668eee2524" path="/var/lib/kubelet/pods/0fe89e6c-771c-4cb7-a4f5-54668eee2524/volumes"
Oct 30 22:15:03.712772 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:03.712758    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dce092df-6490-433b-a5fa-02df7db0b5a8" path="/var/lib/kubelet/pods/dce092df-6490-433b-a5fa-02df7db0b5a8/volumes"
Oct 30 22:15:04.091380 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:04.090821    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l" event={"ID":"1b01d57d-4936-4e45-8b9e-c9ea8e7246fc","Type":"ContainerStarted","Data":"c51143939610487533d43f97dbadc5934dc29c74ddd6661d21642783cf7ddf1c"}
Oct 30 22:15:04.116079 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:04.113146    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt" event={"ID":"830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca","Type":"ContainerStarted","Data":"73593e7f605a5dfeb676c6c8c3905d11429670b4fdace70e47d0eaf607d89df6"}
Oct 30 22:15:04.121678 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:04.120532    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj" event={"ID":"1b48210a-16e2-423e-a233-0d340be66ff5","Type":"ContainerStarted","Data":"61ca35060f750590f6ea556c9074300e64c255bb701d33cba8ce299489bd1258"}
Oct 30 22:15:05.129246 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:05.129211    5087 generic.go:356] "Generic (PLEG): container finished" podID="1b01d57d-4936-4e45-8b9e-c9ea8e7246fc" containerID="c51143939610487533d43f97dbadc5934dc29c74ddd6661d21642783cf7ddf1c" exitCode=60
Oct 30 22:15:05.129750 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:05.129288    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l" event={"ID":"1b01d57d-4936-4e45-8b9e-c9ea8e7246fc","Type":"ContainerDied","Data":"c51143939610487533d43f97dbadc5934dc29c74ddd6661d21642783cf7ddf1c"}
Oct 30 22:15:05.129750 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:05.129375    5087 scope.go:122] "RemoveContainer" containerID="4c42d4488b743b312ba352a4222dcf88347b52c3f853921277ae3a5a37931fc9"
Oct 30 22:15:05.129750 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:05.129639    5087 scope.go:122] "RemoveContainer" containerID="c51143939610487533d43f97dbadc5934dc29c74ddd6661d21642783cf7ddf1c"
Oct 30 22:15:05.130000 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:15:05.129980    5087 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l_odh-ai-gateway-infra(1b01d57d-4936-4e45-8b9e-c9ea8e7246fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l" podUID="1b01d57d-4936-4e45-8b9e-c9ea8e7246fc"
Oct 30 22:15:05.131330 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:05.131310    5087 generic.go:356] "Generic (PLEG): container finished" podID="830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca" containerID="73593e7f605a5dfeb676c6c8c3905d11429670b4fdace70e47d0eaf607d89df6" exitCode=60
Oct 30 22:15:05.131474 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:05.131395    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt" event={"ID":"830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca","Type":"ContainerDied","Data":"73593e7f605a5dfeb676c6c8c3905d11429670b4fdace70e47d0eaf607d89df6"}
Oct 30 22:15:05.131660 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:05.131645    5087 scope.go:122] "RemoveContainer" containerID="73593e7f605a5dfeb676c6c8c3905d11429670b4fdace70e47d0eaf607d89df6"
Oct 30 22:15:05.131925 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:15:05.131904    5087 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt_odh-ai-gateway-infra(830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt" podUID="830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca"
Oct 30 22:15:05.133402 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:05.133380    5087 generic.go:356] "Generic (PLEG): container finished" podID="1b48210a-16e2-423e-a233-0d340be66ff5" containerID="61ca35060f750590f6ea556c9074300e64c255bb701d33cba8ce299489bd1258" exitCode=60
Oct 30 22:15:05.133495 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:05.133420    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj" event={"ID":"1b48210a-16e2-423e-a233-0d340be66ff5","Type":"ContainerDied","Data":"61ca35060f750590f6ea556c9074300e64c255bb701d33cba8ce299489bd1258"}
Oct 30 22:15:05.133659 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:05.133645    5087 scope.go:122] "RemoveContainer" containerID="61ca35060f750590f6ea556c9074300e64c255bb701d33cba8ce299489bd1258"
Oct 30 22:15:05.133887 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:15:05.133870    5087 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj_odh-ai-gateway-infra(1b48210a-16e2-423e-a233-0d340be66ff5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj" podUID="1b48210a-16e2-423e-a233-0d340be66ff5"
Oct 30 22:15:05.142366 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:05.142337    5087 scope.go:122] "RemoveContainer" containerID="241676cf7c59b17e85b6af34d069bc90a05c7f18642c59798cbd771d6611f9c9"
Oct 30 22:15:05.155056 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:05.155036    5087 scope.go:122] "RemoveContainer" containerID="0846efd57e3c41d47dd615e6b369723e06419739c448e824660d0a1ea74a8ef3"
Oct 30 22:15:06.149898 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:06.149857    5087 scope.go:122] "RemoveContainer" containerID="61ca35060f750590f6ea556c9074300e64c255bb701d33cba8ce299489bd1258"
Oct 30 22:15:06.150400 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:15:06.150171    5087 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj_odh-ai-gateway-infra(1b48210a-16e2-423e-a233-0d340be66ff5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj" podUID="1b48210a-16e2-423e-a233-0d340be66ff5"
Oct 30 22:15:06.151755 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:06.151735    5087 scope.go:122] "RemoveContainer" containerID="c51143939610487533d43f97dbadc5934dc29c74ddd6661d21642783cf7ddf1c"
Oct 30 22:15:06.152030 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:15:06.152005    5087 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l_odh-ai-gateway-infra(1b01d57d-4936-4e45-8b9e-c9ea8e7246fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l" podUID="1b01d57d-4936-4e45-8b9e-c9ea8e7246fc"
Oct 30 22:15:06.153340 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:06.153321    5087 scope.go:122] "RemoveContainer" containerID="73593e7f605a5dfeb676c6c8c3905d11429670b4fdace70e47d0eaf607d89df6"
Oct 30 22:15:06.153542 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:15:06.153523    5087 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt_odh-ai-gateway-infra(830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt" podUID="830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca"
Oct 30 22:15:11.168629 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:11.168591    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"]
Oct 30 22:15:11.190062 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:11.190020    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c" podUID="c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d" containerName="istio-proxy" containerID="cri-o://6096c0cb37cd14bf780ed09175ea098d94b23127b945eb763680406d75afcd51" gracePeriod=30
Oct 30 22:15:15.707540 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:15.707504    5087 scope.go:122] "RemoveContainer" containerID="61ca35060f750590f6ea556c9074300e64c255bb701d33cba8ce299489bd1258"
Oct 30 22:15:16.221716 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.221682    5087 patch_prober.go:29] interesting pod/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 30 22:15:16.221858 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.221735    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c" podUID="c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 30 22:15:16.229917 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.229882    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj" event={"ID":"1b48210a-16e2-423e-a233-0d340be66ff5","Type":"ContainerStarted","Data":"36d23cc9a1882ff02284ef611260fccc78b0844b8072b998b43c0e767d4a0028"}
Oct 30 22:15:16.477364 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.477316    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 22:15:16.520177 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.520146    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istiod-ca-cert\") pod \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") "
Oct 30 22:15:16.520371 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.520206    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istio-data\") pod \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") "
Oct 30 22:15:16.520371 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.520254    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-workload-socket\") pod \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") "
Oct 30 22:15:16.520526 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.520375    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-kube-api-access-7q86d\" (UniqueName: \"kubernetes.io/projected/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-kube-api-access-7q86d\") pod \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") "
Oct 30 22:15:16.520526 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.520408    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istio-podinfo\") pod \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") "
Oct 30 22:15:16.520526 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.520485    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istio-token\" (UniqueName: \"kubernetes.io/projected/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istio-token\") pod \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") "
Oct 30 22:15:16.520526 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.520515    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istio-envoy\") pod \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") "
Oct 30 22:15:16.520771 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.520543    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-workload-certs\") pod \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") "
Oct 30 22:15:16.520771 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.520573    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istiod-ca-cert" pod "c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d" (UID: "c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:16.520771 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.520596    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-credential-socket\") pod \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\" (UID: \"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d\") "
Oct 30 22:15:16.520926 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.520828    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istio-data" pod "c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d" (UID: "c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:16.520926 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.520897    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-workload-socket" pod "c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d" (UID: "c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:16.521120 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.521097    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:16.521190 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.521120    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:16.521190 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.521135    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:16.521190 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.521134    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-workload-certs" pod "c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d" (UID: "c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:16.521314 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.521187    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-credential-socket" pod "c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d" (UID: "c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:16.523266 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.523237    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istio-envoy" pod "c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d" (UID: "c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:16.523408 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.523388    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-kube-api-access-7q86d" pod "c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d" (UID: "c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d"). InnerVolumeSpecName "kube-api-access-7q86d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:16.523801 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.523783    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istio-podinfo" pod "c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d" (UID: "c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:15:16.523891 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.523874    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istio-token" pod "c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d" (UID: "c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:16.622392 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.622293    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q86d\" (UniqueName: \"kubernetes.io/projected/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-kube-api-access-7q86d\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:16.622392 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.622333    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:16.622392 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.622366    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:16.622392 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.622379    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:16.622392 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.622393    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:16.622669 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.622407    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:16.702418 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.702377    5087 scope.go:122] "RemoveContainer" containerID="73593e7f605a5dfeb676c6c8c3905d11429670b4fdace70e47d0eaf607d89df6"
Oct 30 22:15:16.702561 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:16.702454    5087 scope.go:122] "RemoveContainer" containerID="c51143939610487533d43f97dbadc5934dc29c74ddd6661d21642783cf7ddf1c"
Oct 30 22:15:17.169836 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:15:17.169800    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:15:17.238608 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:17.238578    5087 generic.go:356] "Generic (PLEG): container finished" podID="1b48210a-16e2-423e-a233-0d340be66ff5" containerID="36d23cc9a1882ff02284ef611260fccc78b0844b8072b998b43c0e767d4a0028" exitCode=60
Oct 30 22:15:17.238811 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:17.238694    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj" event={"ID":"1b48210a-16e2-423e-a233-0d340be66ff5","Type":"ContainerDied","Data":"36d23cc9a1882ff02284ef611260fccc78b0844b8072b998b43c0e767d4a0028"}
Oct 30 22:15:17.238811 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:17.238746    5087 scope.go:122] "RemoveContainer" containerID="61ca35060f750590f6ea556c9074300e64c255bb701d33cba8ce299489bd1258"
Oct 30 22:15:17.239164 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:17.239147    5087 scope.go:122] "RemoveContainer" containerID="36d23cc9a1882ff02284ef611260fccc78b0844b8072b998b43c0e767d4a0028"
Oct 30 22:15:17.239509 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:15:17.239486    5087 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj_odh-ai-gateway-infra(1b48210a-16e2-423e-a233-0d340be66ff5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj" podUID="1b48210a-16e2-423e-a233-0d340be66ff5"
Oct 30 22:15:17.242725 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:17.241485    5087 generic.go:356] "Generic (PLEG): container finished" podID="c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d" containerID="6096c0cb37cd14bf780ed09175ea098d94b23127b945eb763680406d75afcd51" exitCode=0
Oct 30 22:15:17.242725 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:17.241537    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"
Oct 30 22:15:17.242725 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:17.241536    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c" event={"ID":"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d","Type":"ContainerDied","Data":"6096c0cb37cd14bf780ed09175ea098d94b23127b945eb763680406d75afcd51"}
Oct 30 22:15:17.242725 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:17.241600    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c" event={"ID":"c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d","Type":"ContainerDied","Data":"553764ee19e2424d7c061141db243e740aef22761e77295f045df1a219e68c08"}
Oct 30 22:15:17.243837 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:17.243812    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l" event={"ID":"1b01d57d-4936-4e45-8b9e-c9ea8e7246fc","Type":"ContainerStarted","Data":"2eac4d247c83059d86a0972c3714be74347c906b2bef6c7764b69295a2b116c5"}
Oct 30 22:15:17.246262 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:17.246241    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt" event={"ID":"830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca","Type":"ContainerStarted","Data":"5c76f76c8e2855afd01db2fe11d4d76cfc5e8a0947744035ef4f6d748ba8006b"}
Oct 30 22:15:17.253802 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:17.253785    5087 scope.go:122] "RemoveContainer" containerID="6096c0cb37cd14bf780ed09175ea098d94b23127b945eb763680406d75afcd51"
Oct 30 22:15:17.265774 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:17.265760    5087 scope.go:122] "RemoveContainer" containerID="6096c0cb37cd14bf780ed09175ea098d94b23127b945eb763680406d75afcd51"
Oct 30 22:15:17.266036 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:15:17.266015    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6096c0cb37cd14bf780ed09175ea098d94b23127b945eb763680406d75afcd51\": container with ID starting with 6096c0cb37cd14bf780ed09175ea098d94b23127b945eb763680406d75afcd51 not found: ID does not exist" containerID="6096c0cb37cd14bf780ed09175ea098d94b23127b945eb763680406d75afcd51"
Oct 30 22:15:17.266103 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:17.266043    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6096c0cb37cd14bf780ed09175ea098d94b23127b945eb763680406d75afcd51"} err="failed to get container status \"6096c0cb37cd14bf780ed09175ea098d94b23127b945eb763680406d75afcd51\": rpc error: code = NotFound desc = could not find container \"6096c0cb37cd14bf780ed09175ea098d94b23127b945eb763680406d75afcd51\": container with ID starting with 6096c0cb37cd14bf780ed09175ea098d94b23127b945eb763680406d75afcd51 not found: ID does not exist"
Oct 30 22:15:17.384015 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:17.383980    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj"]
Oct 30 22:15:17.667862 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:17.667829    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"]
Oct 30 22:15:17.678928 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:17.678900    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-c3e2d8-openshift-default-5b5d87968c-xb49c"]
Oct 30 22:15:17.709564 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:17.709536    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d" path="/var/lib/kubelet/pods/c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d/volumes"
Oct 30 22:15:18.263716 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:18.263685    5087 generic.go:356] "Generic (PLEG): container finished" podID="1b01d57d-4936-4e45-8b9e-c9ea8e7246fc" containerID="2eac4d247c83059d86a0972c3714be74347c906b2bef6c7764b69295a2b116c5" exitCode=60
Oct 30 22:15:18.264102 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:18.263757    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l" event={"ID":"1b01d57d-4936-4e45-8b9e-c9ea8e7246fc","Type":"ContainerDied","Data":"2eac4d247c83059d86a0972c3714be74347c906b2bef6c7764b69295a2b116c5"}
Oct 30 22:15:18.264102 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:18.263821    5087 scope.go:122] "RemoveContainer" containerID="c51143939610487533d43f97dbadc5934dc29c74ddd6661d21642783cf7ddf1c"
Oct 30 22:15:18.264233 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:18.264178    5087 scope.go:122] "RemoveContainer" containerID="2eac4d247c83059d86a0972c3714be74347c906b2bef6c7764b69295a2b116c5"
Oct 30 22:15:18.264563 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:15:18.264534    5087 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l_odh-ai-gateway-infra(1b01d57d-4936-4e45-8b9e-c9ea8e7246fc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l" podUID="1b01d57d-4936-4e45-8b9e-c9ea8e7246fc"
Oct 30 22:15:18.267596 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:18.267575    5087 generic.go:356] "Generic (PLEG): container finished" podID="830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca" containerID="5c76f76c8e2855afd01db2fe11d4d76cfc5e8a0947744035ef4f6d748ba8006b" exitCode=60
Oct 30 22:15:18.267686 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:18.267641    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt" event={"ID":"830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca","Type":"ContainerDied","Data":"5c76f76c8e2855afd01db2fe11d4d76cfc5e8a0947744035ef4f6d748ba8006b"}
Oct 30 22:15:18.268046 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:18.268030    5087 scope.go:122] "RemoveContainer" containerID="5c76f76c8e2855afd01db2fe11d4d76cfc5e8a0947744035ef4f6d748ba8006b"
Oct 30 22:15:18.268442 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:15:18.268419    5087 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt_odh-ai-gateway-infra(830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt" podUID="830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca"
Oct 30 22:15:18.281181 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:18.281150    5087 scope.go:122] "RemoveContainer" containerID="73593e7f605a5dfeb676c6c8c3905d11429670b4fdace70e47d0eaf607d89df6"
Oct 30 22:15:18.421143 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:18.421117    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj"
Oct 30 22:15:18.441680 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:18.441656    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b48210a-16e2-423e-a233-0d340be66ff5-kube-api-access-fb54j\" (UniqueName: \"kubernetes.io/projected/1b48210a-16e2-423e-a233-0d340be66ff5-kube-api-access-fb54j\") pod \"1b48210a-16e2-423e-a233-0d340be66ff5\" (UID: \"1b48210a-16e2-423e-a233-0d340be66ff5\") "
Oct 30 22:15:18.441850 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:18.441803    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b48210a-16e2-423e-a233-0d340be66ff5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b48210a-16e2-423e-a233-0d340be66ff5-maas-api-service-ca\") pod \"1b48210a-16e2-423e-a233-0d340be66ff5\" (UID: \"1b48210a-16e2-423e-a233-0d340be66ff5\") "
Oct 30 22:15:18.442223 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:18.442193    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b48210a-16e2-423e-a233-0d340be66ff5-maas-api-service-ca" pod "1b48210a-16e2-423e-a233-0d340be66ff5" (UID: "1b48210a-16e2-423e-a233-0d340be66ff5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:18.444007 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:18.443978    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b48210a-16e2-423e-a233-0d340be66ff5-kube-api-access-fb54j" pod "1b48210a-16e2-423e-a233-0d340be66ff5" (UID: "1b48210a-16e2-423e-a233-0d340be66ff5"). InnerVolumeSpecName "kube-api-access-fb54j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:18.543627 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:18.543537    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb54j\" (UniqueName: \"kubernetes.io/projected/1b48210a-16e2-423e-a233-0d340be66ff5-kube-api-access-fb54j\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:18.543627 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:18.543574    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b48210a-16e2-423e-a233-0d340be66ff5-maas-api-service-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:18.555371 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:18.555322    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l"]
Oct 30 22:15:18.801070 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:18.800983    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt"]
Oct 30 22:15:19.280695 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:19.280661    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj"
Oct 30 22:15:19.281086 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:19.280711    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj" event={"ID":"1b48210a-16e2-423e-a233-0d340be66ff5","Type":"ContainerDied","Data":"b84437fc98c6e3e25417fc625ee0b80a104a103a7980fe327bb3b1fff6f35a81"}
Oct 30 22:15:19.281086 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:19.280780    5087 scope.go:122] "RemoveContainer" containerID="36d23cc9a1882ff02284ef611260fccc78b0844b8072b998b43c0e767d4a0028"
Oct 30 22:15:19.447675 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:19.447649    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l"
Oct 30 22:15:19.481268 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:19.481245    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt"
Oct 30 22:15:19.523316 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:19.523286    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj"]
Oct 30 22:15:19.530975 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:19.530913    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-bc009f-29889975-hlvrj"]
Oct 30 22:15:19.556240 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:19.556212    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b01d57d-4936-4e45-8b9e-c9ea8e7246fc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b01d57d-4936-4e45-8b9e-c9ea8e7246fc-maas-api-service-ca\") pod \"1b01d57d-4936-4e45-8b9e-c9ea8e7246fc\" (UID: \"1b01d57d-4936-4e45-8b9e-c9ea8e7246fc\") "
Oct 30 22:15:19.556375 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:19.556306    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b01d57d-4936-4e45-8b9e-c9ea8e7246fc-kube-api-access-phgrb\" (UniqueName: \"kubernetes.io/projected/1b01d57d-4936-4e45-8b9e-c9ea8e7246fc-kube-api-access-phgrb\") pod \"1b01d57d-4936-4e45-8b9e-c9ea8e7246fc\" (UID: \"1b01d57d-4936-4e45-8b9e-c9ea8e7246fc\") "
Oct 30 22:15:19.556429 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:19.556401    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca-maas-api-service-ca\") pod \"830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca\" (UID: \"830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca\") "
Oct 30 22:15:19.556544 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:19.556438    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca-kube-api-access-rm4gh\" (UniqueName: \"kubernetes.io/projected/830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca-kube-api-access-rm4gh\") pod \"830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca\" (UID: \"830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca\") "
Oct 30 22:15:19.556610 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:19.556540    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b01d57d-4936-4e45-8b9e-c9ea8e7246fc-maas-api-service-ca" pod "1b01d57d-4936-4e45-8b9e-c9ea8e7246fc" (UID: "1b01d57d-4936-4e45-8b9e-c9ea8e7246fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:19.556844 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:19.556821    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca-maas-api-service-ca" pod "830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca" (UID: "830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:19.556913 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:19.556852    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b01d57d-4936-4e45-8b9e-c9ea8e7246fc-maas-api-service-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.558581 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:19.558558    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b01d57d-4936-4e45-8b9e-c9ea8e7246fc-kube-api-access-phgrb" pod "1b01d57d-4936-4e45-8b9e-c9ea8e7246fc" (UID: "1b01d57d-4936-4e45-8b9e-c9ea8e7246fc"). InnerVolumeSpecName "kube-api-access-phgrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:19.558676 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:19.558589    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca-kube-api-access-rm4gh" pod "830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca" (UID: "830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca"). InnerVolumeSpecName "kube-api-access-rm4gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:19.663393 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:19.659717    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm4gh\" (UniqueName: \"kubernetes.io/projected/830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca-kube-api-access-rm4gh\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.663393 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:19.659757    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phgrb\" (UniqueName: \"kubernetes.io/projected/1b01d57d-4936-4e45-8b9e-c9ea8e7246fc-kube-api-access-phgrb\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.663393 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:19.659773    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca-maas-api-service-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.708812 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:19.708779    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b48210a-16e2-423e-a233-0d340be66ff5" path="/var/lib/kubelet/pods/1b48210a-16e2-423e-a233-0d340be66ff5/volumes"
Oct 30 22:15:20.291795 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:20.291763    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l"
Oct 30 22:15:20.292334 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:20.291767    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l" event={"ID":"1b01d57d-4936-4e45-8b9e-c9ea8e7246fc","Type":"ContainerDied","Data":"79c61e2ad5c4ba433d0a86f9032761114c3ee533871edc5ad4402d0baf93b3a7"}
Oct 30 22:15:20.292334 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:20.292017    5087 scope.go:122] "RemoveContainer" containerID="2eac4d247c83059d86a0972c3714be74347c906b2bef6c7764b69295a2b116c5"
Oct 30 22:15:20.295228 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:20.294763    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt"
Oct 30 22:15:20.296222 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:20.294634    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt" event={"ID":"830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca","Type":"ContainerDied","Data":"95d66a5d679969d667a8070fefdedf27f149e2018cc934330cff31c1f3cd935c"}
Oct 30 22:15:20.306867 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:20.306845    5087 scope.go:122] "RemoveContainer" containerID="5c76f76c8e2855afd01db2fe11d4d76cfc5e8a0947744035ef4f6d748ba8006b"
Oct 30 22:15:20.466497 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:20.466459    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt"]
Oct 30 22:15:20.471236 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:20.471207    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-9da607-29889975-mp5bt"]
Oct 30 22:15:20.585437 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:20.578248    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l"]
Oct 30 22:15:20.585437 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:20.583072    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-954628-29889975-tvr8l"]
Oct 30 22:15:21.708458 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:21.708429    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b01d57d-4936-4e45-8b9e-c9ea8e7246fc" path="/var/lib/kubelet/pods/1b01d57d-4936-4e45-8b9e-c9ea8e7246fc/volumes"
Oct 30 22:15:21.708904 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:21.708889    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca" path="/var/lib/kubelet/pods/830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca/volumes"
Oct 30 22:15:29.525054 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:29.525025    5087 scope.go:122] "RemoveContainer" containerID="ff8a49b8661b2f37638305ae5f35e5b2c82fc37011f2e23d4ac005a2f4e83b18"
Oct 30 22:15:29.534951 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:29.534930    5087 scope.go:122] "RemoveContainer" containerID="cd2e69f347fc6ebeea80caac1a39009db474d9edcc0145d5bcd9da1f22c5db5a"
Oct 30 22:15:29.544085 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:29.544062    5087 scope.go:122] "RemoveContainer" containerID="f890fdea7e844b91994f604ebe7fc3a8714cf38278576f8d44893f2d3210c542"
Oct 30 22:15:29.552636 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:29.552612    5087 scope.go:122] "RemoveContainer" containerID="9f54313622fcdc7445e121626a9383a51729f49ea1b17024116cf7eba9bddf73"
Oct 30 22:15:31.110202 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:31.110169    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"]
Oct 30 22:15:31.110602 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:31.110404    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm" podUID="cf713fc3-b05d-453b-8e78-598e256b1e1b" containerName="istio-proxy" containerID="cri-o://db81f7d4f8f112846732de3cd8924bb09227010a0e96dcce90761a10b9235256" gracePeriod=30
Oct 30 22:15:31.150584 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:31.150552    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"]
Oct 30 22:15:31.150853 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:31.150802    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54" podUID="53f51d9f-f90e-43f9-85de-eab0d05086ee" containerName="istio-proxy" containerID="cri-o://c5bc22f224b29c496d2a94ba40586fa6be2a0e0e05eaa85da39ac6989a2c6e94" gracePeriod=30
Oct 30 22:15:31.153404 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:31.153359    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"]
Oct 30 22:15:31.153622 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:31.153578    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq" podUID="5db02748-f530-4034-b739-e9b4b9a263d5" containerName="istio-proxy" containerID="cri-o://c85d8495b561e86ebe802aa367174df9e8391ac0a562ee115b1cb66439cdc08e" gracePeriod=30
Oct 30 22:15:33.445758 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:33.445725    5087 patch_prober.go:29] interesting pod/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:15:33.446236 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:33.445786    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq" podUID="5db02748-f530-4034-b739-e9b4b9a263d5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:15:34.219919 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:34.219885    5087 patch_prober.go:29] interesting pod/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:15:34.220096 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:34.219941    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54" podUID="53f51d9f-f90e-43f9-85de-eab0d05086ee" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:15:34.393150 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:34.393109    5087 patch_prober.go:29] interesting pod/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:15:34.393328 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:34.393168    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm" podUID="cf713fc3-b05d-453b-8e78-598e256b1e1b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:15:36.417663 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.417487    5087 generic.go:356] "Generic (PLEG): container finished" podID="cf713fc3-b05d-453b-8e78-598e256b1e1b" containerID="db81f7d4f8f112846732de3cd8924bb09227010a0e96dcce90761a10b9235256" exitCode=0
Oct 30 22:15:36.417663 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.417599    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm" event={"ID":"cf713fc3-b05d-453b-8e78-598e256b1e1b","Type":"ContainerDied","Data":"db81f7d4f8f112846732de3cd8924bb09227010a0e96dcce90761a10b9235256"}
Oct 30 22:15:36.420598 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.420319    5087 generic.go:356] "Generic (PLEG): container finished" podID="53f51d9f-f90e-43f9-85de-eab0d05086ee" containerID="c5bc22f224b29c496d2a94ba40586fa6be2a0e0e05eaa85da39ac6989a2c6e94" exitCode=0
Oct 30 22:15:36.420598 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.420532    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54" event={"ID":"53f51d9f-f90e-43f9-85de-eab0d05086ee","Type":"ContainerDied","Data":"c5bc22f224b29c496d2a94ba40586fa6be2a0e0e05eaa85da39ac6989a2c6e94"}
Oct 30 22:15:36.422870 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.422713    5087 generic.go:356] "Generic (PLEG): container finished" podID="5db02748-f530-4034-b739-e9b4b9a263d5" containerID="c85d8495b561e86ebe802aa367174df9e8391ac0a562ee115b1cb66439cdc08e" exitCode=0
Oct 30 22:15:36.422870 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.422819    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq" event={"ID":"5db02748-f530-4034-b739-e9b4b9a263d5","Type":"ContainerDied","Data":"c85d8495b561e86ebe802aa367174df9e8391ac0a562ee115b1cb66439cdc08e"}
Oct 30 22:15:36.457156 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.456892    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:15:36.502628 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.502591    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:15:36.512952 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.512922    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:15:36.557114 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.557080    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-credential-socket\") pod \"cf713fc3-b05d-453b-8e78-598e256b1e1b\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") "
Oct 30 22:15:36.557114 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.557122    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf713fc3-b05d-453b-8e78-598e256b1e1b-istio-token\" (UniqueName: \"kubernetes.io/projected/cf713fc3-b05d-453b-8e78-598e256b1e1b-istio-token\") pod \"cf713fc3-b05d-453b-8e78-598e256b1e1b\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") "
Oct 30 22:15:36.557407 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.557226    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-istio-data\") pod \"cf713fc3-b05d-453b-8e78-598e256b1e1b\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") "
Oct 30 22:15:36.557407 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.557293    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-workload-certs\") pod \"cf713fc3-b05d-453b-8e78-598e256b1e1b\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") "
Oct 30 22:15:36.557407 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.557328    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-istio-envoy\") pod \"cf713fc3-b05d-453b-8e78-598e256b1e1b\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") "
Oct 30 22:15:36.557407 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.557395    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-workload-socket\") pod \"cf713fc3-b05d-453b-8e78-598e256b1e1b\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") "
Oct 30 22:15:36.557634 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.557455    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-credential-socket" pod "cf713fc3-b05d-453b-8e78-598e256b1e1b" (UID: "cf713fc3-b05d-453b-8e78-598e256b1e1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:36.557634 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.557467    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf713fc3-b05d-453b-8e78-598e256b1e1b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf713fc3-b05d-453b-8e78-598e256b1e1b-istiod-ca-cert\") pod \"cf713fc3-b05d-453b-8e78-598e256b1e1b\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") "
Oct 30 22:15:36.557634 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.557482    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-istio-data" pod "cf713fc3-b05d-453b-8e78-598e256b1e1b" (UID: "cf713fc3-b05d-453b-8e78-598e256b1e1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:36.557634 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.557511    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf713fc3-b05d-453b-8e78-598e256b1e1b-kube-api-access-6jc56\" (UniqueName: \"kubernetes.io/projected/cf713fc3-b05d-453b-8e78-598e256b1e1b-kube-api-access-6jc56\") pod \"cf713fc3-b05d-453b-8e78-598e256b1e1b\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") "
Oct 30 22:15:36.557634 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.557533    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf713fc3-b05d-453b-8e78-598e256b1e1b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf713fc3-b05d-453b-8e78-598e256b1e1b-istio-podinfo\") pod \"cf713fc3-b05d-453b-8e78-598e256b1e1b\" (UID: \"cf713fc3-b05d-453b-8e78-598e256b1e1b\") "
Oct 30 22:15:36.557945 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.557917    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf713fc3-b05d-453b-8e78-598e256b1e1b-istiod-ca-cert" pod "cf713fc3-b05d-453b-8e78-598e256b1e1b" (UID: "cf713fc3-b05d-453b-8e78-598e256b1e1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:36.558078 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.558026    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf713fc3-b05d-453b-8e78-598e256b1e1b-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.558078 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.558046    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.558078 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.558060    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.558339 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.558206    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-workload-socket" pod "cf713fc3-b05d-453b-8e78-598e256b1e1b" (UID: "cf713fc3-b05d-453b-8e78-598e256b1e1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:36.558448 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.558302    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-workload-certs" pod "cf713fc3-b05d-453b-8e78-598e256b1e1b" (UID: "cf713fc3-b05d-453b-8e78-598e256b1e1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:36.559776 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.559755    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-istio-envoy" pod "cf713fc3-b05d-453b-8e78-598e256b1e1b" (UID: "cf713fc3-b05d-453b-8e78-598e256b1e1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:36.560140 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.560119    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf713fc3-b05d-453b-8e78-598e256b1e1b-istio-podinfo" pod "cf713fc3-b05d-453b-8e78-598e256b1e1b" (UID: "cf713fc3-b05d-453b-8e78-598e256b1e1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:15:36.560231 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.560153    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf713fc3-b05d-453b-8e78-598e256b1e1b-istio-token" pod "cf713fc3-b05d-453b-8e78-598e256b1e1b" (UID: "cf713fc3-b05d-453b-8e78-598e256b1e1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:36.560530 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.560512    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf713fc3-b05d-453b-8e78-598e256b1e1b-kube-api-access-6jc56" pod "cf713fc3-b05d-453b-8e78-598e256b1e1b" (UID: "cf713fc3-b05d-453b-8e78-598e256b1e1b"). InnerVolumeSpecName "kube-api-access-6jc56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:36.658732 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.658697    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-istio-envoy\") pod \"5db02748-f530-4034-b739-e9b4b9a263d5\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") "
Oct 30 22:15:36.658926 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.658757    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5db02748-f530-4034-b739-e9b4b9a263d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db02748-f530-4034-b739-e9b4b9a263d5-istiod-ca-cert\") pod \"5db02748-f530-4034-b739-e9b4b9a263d5\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") "
Oct 30 22:15:36.658926 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.658796    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-workload-socket\") pod \"5db02748-f530-4034-b739-e9b4b9a263d5\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") "
Oct 30 22:15:36.658926 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.658838    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5db02748-f530-4034-b739-e9b4b9a263d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db02748-f530-4034-b739-e9b4b9a263d5-istio-podinfo\") pod \"5db02748-f530-4034-b739-e9b4b9a263d5\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") "
Oct 30 22:15:36.658926 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.658872    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53f51d9f-f90e-43f9-85de-eab0d05086ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53f51d9f-f90e-43f9-85de-eab0d05086ee-istiod-ca-cert\") pod \"53f51d9f-f90e-43f9-85de-eab0d05086ee\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") "
Oct 30 22:15:36.658926 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.658911    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-workload-certs\") pod \"5db02748-f530-4034-b739-e9b4b9a263d5\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") "
Oct 30 22:15:36.659228 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.658995    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-istio-data\") pod \"5db02748-f530-4034-b739-e9b4b9a263d5\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") "
Oct 30 22:15:36.659228 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.659051    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53f51d9f-f90e-43f9-85de-eab0d05086ee-kube-api-access-h5cg4\" (UniqueName: \"kubernetes.io/projected/53f51d9f-f90e-43f9-85de-eab0d05086ee-kube-api-access-h5cg4\") pod \"53f51d9f-f90e-43f9-85de-eab0d05086ee\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") "
Oct 30 22:15:36.659228 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.659127    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53f51d9f-f90e-43f9-85de-eab0d05086ee-istio-token\" (UniqueName: \"kubernetes.io/projected/53f51d9f-f90e-43f9-85de-eab0d05086ee-istio-token\") pod \"53f51d9f-f90e-43f9-85de-eab0d05086ee\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") "
Oct 30 22:15:36.659228 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.659155    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db02748-f530-4034-b739-e9b4b9a263d5-kube-api-access-wxvtk\" (UniqueName: \"kubernetes.io/projected/5db02748-f530-4034-b739-e9b4b9a263d5-kube-api-access-wxvtk\") pod \"5db02748-f530-4034-b739-e9b4b9a263d5\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") "
Oct 30 22:15:36.659228 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.659220    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-credential-socket\") pod \"53f51d9f-f90e-43f9-85de-eab0d05086ee\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") "
Oct 30 22:15:36.659578 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.659218    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-workload-socket" pod "5db02748-f530-4034-b739-e9b4b9a263d5" (UID: "5db02748-f530-4034-b739-e9b4b9a263d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:36.659578 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.659258    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-workload-socket\") pod \"53f51d9f-f90e-43f9-85de-eab0d05086ee\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") "
Oct 30 22:15:36.659578 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.659258    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-workload-certs" pod "5db02748-f530-4034-b739-e9b4b9a263d5" (UID: "5db02748-f530-4034-b739-e9b4b9a263d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:36.659578 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.659286    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db02748-f530-4034-b739-e9b4b9a263d5-istio-token\" (UniqueName: \"kubernetes.io/projected/5db02748-f530-4034-b739-e9b4b9a263d5-istio-token\") pod \"5db02748-f530-4034-b739-e9b4b9a263d5\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") "
Oct 30 22:15:36.659578 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.659329    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-credential-socket\") pod \"5db02748-f530-4034-b739-e9b4b9a263d5\" (UID: \"5db02748-f530-4034-b739-e9b4b9a263d5\") "
Oct 30 22:15:36.659578 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.659414    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-istio-data\") pod \"53f51d9f-f90e-43f9-85de-eab0d05086ee\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") "
Oct 30 22:15:36.659578 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.659462    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-workload-certs\") pod \"53f51d9f-f90e-43f9-85de-eab0d05086ee\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") "
Oct 30 22:15:36.659578 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.659491    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-istio-envoy\") pod \"53f51d9f-f90e-43f9-85de-eab0d05086ee\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") "
Oct 30 22:15:36.659578 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.659533    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53f51d9f-f90e-43f9-85de-eab0d05086ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53f51d9f-f90e-43f9-85de-eab0d05086ee-istio-podinfo\") pod \"53f51d9f-f90e-43f9-85de-eab0d05086ee\" (UID: \"53f51d9f-f90e-43f9-85de-eab0d05086ee\") "
Oct 30 22:15:36.659578 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.659605    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53f51d9f-f90e-43f9-85de-eab0d05086ee-istiod-ca-cert" pod "53f51d9f-f90e-43f9-85de-eab0d05086ee" (UID: "53f51d9f-f90e-43f9-85de-eab0d05086ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:36.660254 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.659968    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-credential-socket" pod "53f51d9f-f90e-43f9-85de-eab0d05086ee" (UID: "53f51d9f-f90e-43f9-85de-eab0d05086ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:36.660254 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.660040    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53f51d9f-f90e-43f9-85de-eab0d05086ee-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.660254 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.660062    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.660254 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.660077    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf713fc3-b05d-453b-8e78-598e256b1e1b-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.660254 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.660092    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.660254 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.660107    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.660254 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.660120    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.660254 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.660133    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf713fc3-b05d-453b-8e78-598e256b1e1b-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.660254 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.660145    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.660254 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.660206    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jc56\" (UniqueName: \"kubernetes.io/projected/cf713fc3-b05d-453b-8e78-598e256b1e1b-kube-api-access-6jc56\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.660254 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.660223    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf713fc3-b05d-453b-8e78-598e256b1e1b-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.660254 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.660154    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-workload-socket" pod "53f51d9f-f90e-43f9-85de-eab0d05086ee" (UID: "53f51d9f-f90e-43f9-85de-eab0d05086ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:36.660919 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.660305    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-istio-data" pod "5db02748-f530-4034-b739-e9b4b9a263d5" (UID: "5db02748-f530-4034-b739-e9b4b9a263d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:36.660919 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.660373    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-istio-data" pod "53f51d9f-f90e-43f9-85de-eab0d05086ee" (UID: "53f51d9f-f90e-43f9-85de-eab0d05086ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:36.660919 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.660446    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-credential-socket" pod "5db02748-f530-4034-b739-e9b4b9a263d5" (UID: "5db02748-f530-4034-b739-e9b4b9a263d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:36.660919 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.660457    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-workload-certs" pod "53f51d9f-f90e-43f9-85de-eab0d05086ee" (UID: "53f51d9f-f90e-43f9-85de-eab0d05086ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:36.661195 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.661008    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db02748-f530-4034-b739-e9b4b9a263d5-istiod-ca-cert" pod "5db02748-f530-4034-b739-e9b4b9a263d5" (UID: "5db02748-f530-4034-b739-e9b4b9a263d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:36.661608 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.661581    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-istio-envoy" pod "5db02748-f530-4034-b739-e9b4b9a263d5" (UID: "5db02748-f530-4034-b739-e9b4b9a263d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:36.664307 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.664287    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5db02748-f530-4034-b739-e9b4b9a263d5-istio-podinfo" pod "5db02748-f530-4034-b739-e9b4b9a263d5" (UID: "5db02748-f530-4034-b739-e9b4b9a263d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:15:36.665904 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.665857    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-istio-envoy" pod "53f51d9f-f90e-43f9-85de-eab0d05086ee" (UID: "53f51d9f-f90e-43f9-85de-eab0d05086ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:15:36.667160 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.667091    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53f51d9f-f90e-43f9-85de-eab0d05086ee-istio-podinfo" pod "53f51d9f-f90e-43f9-85de-eab0d05086ee" (UID: "53f51d9f-f90e-43f9-85de-eab0d05086ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:15:36.667738 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.667714    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db02748-f530-4034-b739-e9b4b9a263d5-istio-token" pod "5db02748-f530-4034-b739-e9b4b9a263d5" (UID: "5db02748-f530-4034-b739-e9b4b9a263d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:36.667835 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.667752    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f51d9f-f90e-43f9-85de-eab0d05086ee-istio-token" pod "53f51d9f-f90e-43f9-85de-eab0d05086ee" (UID: "53f51d9f-f90e-43f9-85de-eab0d05086ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:36.667903 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.667861    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db02748-f530-4034-b739-e9b4b9a263d5-kube-api-access-wxvtk" pod "5db02748-f530-4034-b739-e9b4b9a263d5" (UID: "5db02748-f530-4034-b739-e9b4b9a263d5"). InnerVolumeSpecName "kube-api-access-wxvtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:36.669074 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.669048    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f51d9f-f90e-43f9-85de-eab0d05086ee-kube-api-access-h5cg4" pod "53f51d9f-f90e-43f9-85de-eab0d05086ee" (UID: "53f51d9f-f90e-43f9-85de-eab0d05086ee"). InnerVolumeSpecName "kube-api-access-h5cg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:36.761810 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.761772    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.761810 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.761810    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db02748-f530-4034-b739-e9b4b9a263d5-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.762089 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.761824    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db02748-f530-4034-b739-e9b4b9a263d5-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.762089 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.761836    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.762089 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.761849    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5cg4\" (UniqueName: \"kubernetes.io/projected/53f51d9f-f90e-43f9-85de-eab0d05086ee-kube-api-access-h5cg4\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.762089 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.761862    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53f51d9f-f90e-43f9-85de-eab0d05086ee-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.762089 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.761874    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxvtk\" (UniqueName: \"kubernetes.io/projected/5db02748-f530-4034-b739-e9b4b9a263d5-kube-api-access-wxvtk\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.762089 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.761887    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.762089 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.761899    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db02748-f530-4034-b739-e9b4b9a263d5-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.762089 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.761911    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db02748-f530-4034-b739-e9b4b9a263d5-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.762089 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.761923    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.762089 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.761936    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.762089 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.761947    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53f51d9f-f90e-43f9-85de-eab0d05086ee-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:36.762089 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:36.761960    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53f51d9f-f90e-43f9-85de-eab0d05086ee-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:15:37.431930 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:37.431850    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"
Oct 30 22:15:37.431930 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:37.431886    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq" event={"ID":"5db02748-f530-4034-b739-e9b4b9a263d5","Type":"ContainerDied","Data":"655ae6ee888d31b458eb93ae21402f5c73a8c1fc124e1e527ca24f493b9eee35"}
Oct 30 22:15:37.432452 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:37.431951    5087 scope.go:122] "RemoveContainer" containerID="c85d8495b561e86ebe802aa367174df9e8391ac0a562ee115b1cb66439cdc08e"
Oct 30 22:15:37.433557 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:37.433437    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"
Oct 30 22:15:37.433557 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:37.433464    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm" event={"ID":"cf713fc3-b05d-453b-8e78-598e256b1e1b","Type":"ContainerDied","Data":"1f4ec5e1df24e843ce1f10d7c53b67a5886641a6dac79900d6fa394ce420162c"}
Oct 30 22:15:37.435171 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:37.435143    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"
Oct 30 22:15:37.435315 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:37.435146    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54" event={"ID":"53f51d9f-f90e-43f9-85de-eab0d05086ee","Type":"ContainerDied","Data":"6baa4ea4027c250b331b46e390f9318ff210da4cb181e5c9dc9d6c0238d7d7a7"}
Oct 30 22:15:37.443018 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:37.442997    5087 scope.go:122] "RemoveContainer" containerID="db81f7d4f8f112846732de3cd8924bb09227010a0e96dcce90761a10b9235256"
Oct 30 22:15:37.453035 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:37.453014    5087 scope.go:122] "RemoveContainer" containerID="c5bc22f224b29c496d2a94ba40586fa6be2a0e0e05eaa85da39ac6989a2c6e94"
Oct 30 22:15:37.598270 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:37.598238    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"]
Oct 30 22:15:37.648770 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:37.648741    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-4kk54"]
Oct 30 22:15:37.710202 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:37.710116    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53f51d9f-f90e-43f9-85de-eab0d05086ee" path="/var/lib/kubelet/pods/53f51d9f-f90e-43f9-85de-eab0d05086ee/volumes"
Oct 30 22:15:37.711816 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:37.711788    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"]
Oct 30 22:15:37.716774 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:37.716750    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-npttq"]
Oct 30 22:15:37.824312 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:37.824277    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"]
Oct 30 22:15:37.827677 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:37.827655    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6df3ba0e-openshift-default-5fdd5fcfbf-wc8sm"]
Oct 30 22:15:39.708434 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:39.708402    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5db02748-f530-4034-b739-e9b4b9a263d5" path="/var/lib/kubelet/pods/5db02748-f530-4034-b739-e9b4b9a263d5/volumes"
Oct 30 22:15:39.708870 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:39.708857    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf713fc3-b05d-453b-8e78-598e256b1e1b" path="/var/lib/kubelet/pods/cf713fc3-b05d-453b-8e78-598e256b1e1b/volumes"
Oct 30 22:15:47.167745 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:15:47.167702    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:15:55.156250 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:55.156201    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"]
Oct 30 22:15:55.156776 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:55.156499    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj" podUID="284b12e8-9e86-4406-8494-360e02159b74" containerName="istio-proxy" containerID="cri-o://07e6718e2230c96163eb10433b31764ff49c04c71a9bd4c4e51ef35518d5a11b" gracePeriod=30
Oct 30 22:15:58.632122 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:58.632090    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c79cb5-464sn"]
Oct 30 22:15:58.683219 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:58.683188    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c79cb5-464sn"]
Oct 30 22:15:59.709139 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:15:59.709105    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f" path="/var/lib/kubelet/pods/7c6aa6a7-616a-49ba-b9b0-edf1d4f8527f/volumes"
Oct 30 22:16:00.420514 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.420454    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 22:16:00.468835 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.468805    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-istio-data\") pod \"284b12e8-9e86-4406-8494-360e02159b74\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") "
Oct 30 22:16:00.468991 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.468868    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/284b12e8-9e86-4406-8494-360e02159b74-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/284b12e8-9e86-4406-8494-360e02159b74-istio-podinfo\") pod \"284b12e8-9e86-4406-8494-360e02159b74\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") "
Oct 30 22:16:00.468991 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.468911    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/284b12e8-9e86-4406-8494-360e02159b74-istio-token\" (UniqueName: \"kubernetes.io/projected/284b12e8-9e86-4406-8494-360e02159b74-istio-token\") pod \"284b12e8-9e86-4406-8494-360e02159b74\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") "
Oct 30 22:16:00.468991 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.468940    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-credential-socket\") pod \"284b12e8-9e86-4406-8494-360e02159b74\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") "
Oct 30 22:16:00.468991 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.468967    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-workload-socket\") pod \"284b12e8-9e86-4406-8494-360e02159b74\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") "
Oct 30 22:16:00.469211 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.469036    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/284b12e8-9e86-4406-8494-360e02159b74-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/284b12e8-9e86-4406-8494-360e02159b74-istiod-ca-cert\") pod \"284b12e8-9e86-4406-8494-360e02159b74\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") "
Oct 30 22:16:00.469211 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.469079    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-istio-envoy\") pod \"284b12e8-9e86-4406-8494-360e02159b74\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") "
Oct 30 22:16:00.469211 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.469086    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-istio-data" pod "284b12e8-9e86-4406-8494-360e02159b74" (UID: "284b12e8-9e86-4406-8494-360e02159b74"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:00.469211 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.469163    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-workload-certs\") pod \"284b12e8-9e86-4406-8494-360e02159b74\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") "
Oct 30 22:16:00.469443 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.469224    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-workload-socket" pod "284b12e8-9e86-4406-8494-360e02159b74" (UID: "284b12e8-9e86-4406-8494-360e02159b74"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:00.469443 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.469270    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/284b12e8-9e86-4406-8494-360e02159b74-kube-api-access-k7m46\" (UniqueName: \"kubernetes.io/projected/284b12e8-9e86-4406-8494-360e02159b74-kube-api-access-k7m46\") pod \"284b12e8-9e86-4406-8494-360e02159b74\" (UID: \"284b12e8-9e86-4406-8494-360e02159b74\") "
Oct 30 22:16:00.469443 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.469335    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-credential-socket" pod "284b12e8-9e86-4406-8494-360e02159b74" (UID: "284b12e8-9e86-4406-8494-360e02159b74"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:00.469585 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.469525    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-workload-certs" pod "284b12e8-9e86-4406-8494-360e02159b74" (UID: "284b12e8-9e86-4406-8494-360e02159b74"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:00.469758 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.469733    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:00.469758 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.469759    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:00.469989 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.469775    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:00.469989 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.469789    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:00.470273 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.470243    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/284b12e8-9e86-4406-8494-360e02159b74-istiod-ca-cert" pod "284b12e8-9e86-4406-8494-360e02159b74" (UID: "284b12e8-9e86-4406-8494-360e02159b74"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:00.471661 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.471609    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/284b12e8-9e86-4406-8494-360e02159b74-kube-api-access-k7m46" pod "284b12e8-9e86-4406-8494-360e02159b74" (UID: "284b12e8-9e86-4406-8494-360e02159b74"). InnerVolumeSpecName "kube-api-access-k7m46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:00.471931 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.471909    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/284b12e8-9e86-4406-8494-360e02159b74-istio-podinfo" pod "284b12e8-9e86-4406-8494-360e02159b74" (UID: "284b12e8-9e86-4406-8494-360e02159b74"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:16:00.472015 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.472001    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-istio-envoy" pod "284b12e8-9e86-4406-8494-360e02159b74" (UID: "284b12e8-9e86-4406-8494-360e02159b74"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:00.472138 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.472123    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/284b12e8-9e86-4406-8494-360e02159b74-istio-token" pod "284b12e8-9e86-4406-8494-360e02159b74" (UID: "284b12e8-9e86-4406-8494-360e02159b74"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:00.570562 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.570526    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7m46\" (UniqueName: \"kubernetes.io/projected/284b12e8-9e86-4406-8494-360e02159b74-kube-api-access-k7m46\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:00.570562 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.570552    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/284b12e8-9e86-4406-8494-360e02159b74-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:00.570562 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.570567    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/284b12e8-9e86-4406-8494-360e02159b74-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:00.570789 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.570577    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/284b12e8-9e86-4406-8494-360e02159b74-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:00.570789 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.570586    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/284b12e8-9e86-4406-8494-360e02159b74-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:00.628298 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.628259    5087 generic.go:356] "Generic (PLEG): container finished" podID="284b12e8-9e86-4406-8494-360e02159b74" containerID="07e6718e2230c96163eb10433b31764ff49c04c71a9bd4c4e51ef35518d5a11b" exitCode=0
Oct 30 22:16:00.628494 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.628297    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj" event={"ID":"284b12e8-9e86-4406-8494-360e02159b74","Type":"ContainerDied","Data":"07e6718e2230c96163eb10433b31764ff49c04c71a9bd4c4e51ef35518d5a11b"}
Oct 30 22:16:00.628494 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.628329    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"
Oct 30 22:16:00.628494 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.628389    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj" event={"ID":"284b12e8-9e86-4406-8494-360e02159b74","Type":"ContainerDied","Data":"dc94bb436e0fbbd68b7849bfda1edc66bf1348e8490f95c0a83d83e636c333e9"}
Oct 30 22:16:00.628494 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.628424    5087 scope.go:122] "RemoveContainer" containerID="07e6718e2230c96163eb10433b31764ff49c04c71a9bd4c4e51ef35518d5a11b"
Oct 30 22:16:00.639310 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.639287    5087 scope.go:122] "RemoveContainer" containerID="07e6718e2230c96163eb10433b31764ff49c04c71a9bd4c4e51ef35518d5a11b"
Oct 30 22:16:00.639729 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:16:00.639705    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07e6718e2230c96163eb10433b31764ff49c04c71a9bd4c4e51ef35518d5a11b\": container with ID starting with 07e6718e2230c96163eb10433b31764ff49c04c71a9bd4c4e51ef35518d5a11b not found: ID does not exist" containerID="07e6718e2230c96163eb10433b31764ff49c04c71a9bd4c4e51ef35518d5a11b"
Oct 30 22:16:00.639841 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.639741    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07e6718e2230c96163eb10433b31764ff49c04c71a9bd4c4e51ef35518d5a11b"} err="failed to get container status \"07e6718e2230c96163eb10433b31764ff49c04c71a9bd4c4e51ef35518d5a11b\": rpc error: code = NotFound desc = could not find container \"07e6718e2230c96163eb10433b31764ff49c04c71a9bd4c4e51ef35518d5a11b\": container with ID starting with 07e6718e2230c96163eb10433b31764ff49c04c71a9bd4c4e51ef35518d5a11b not found: ID does not exist"
Oct 30 22:16:00.793044 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.793005    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"]
Oct 30 22:16:00.799033 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:00.799008    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-bc009f-openshift-default-645d5cbf67-lg5tj"]
Oct 30 22:16:01.708643 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:01.708613    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="284b12e8-9e86-4406-8494-360e02159b74" path="/var/lib/kubelet/pods/284b12e8-9e86-4406-8494-360e02159b74/volumes"
Oct 30 22:16:05.244617 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:05.244520    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"]
Oct 30 22:16:05.245181 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:05.244855    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l" podUID="b9f7c360-2a81-4e01-b104-6e9907c6cb39" containerName="istio-proxy" containerID="cri-o://425a4c700a055308a86b8dd186fbb8ae7514dfc3e3c1c17713e941a2f54c5aaf" gracePeriod=30
Oct 30 22:16:06.593516 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:06.593476    5087 patch_prober.go:29] interesting pod/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:16:06.593945 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:06.593546    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l" podUID="b9f7c360-2a81-4e01-b104-6e9907c6cb39" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:16:08.990837 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.990797    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-632149ee-hd66x"]
Oct 30 22:16:08.991270 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991261    5087 cpu_manager.go:414] "containerMap: removing container" podUID="284b12e8-9e86-4406-8494-360e02159b74" containerName="istio-proxy"
Oct 30 22:16:08.991326 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991274    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="284b12e8-9e86-4406-8494-360e02159b74" containerName="istio-proxy"
Oct 30 22:16:08.991326 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991287    5087 cpu_manager.go:414] "containerMap: removing container" podUID="1b01d57d-4936-4e45-8b9e-c9ea8e7246fc" containerName="cleanup"
Oct 30 22:16:08.991326 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991292    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b01d57d-4936-4e45-8b9e-c9ea8e7246fc" containerName="cleanup"
Oct 30 22:16:08.991326 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991299    5087 cpu_manager.go:414] "containerMap: removing container" podUID="1b48210a-16e2-423e-a233-0d340be66ff5" containerName="cleanup"
Oct 30 22:16:08.991326 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991304    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b48210a-16e2-423e-a233-0d340be66ff5" containerName="cleanup"
Oct 30 22:16:08.991326 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991314    5087 cpu_manager.go:414] "containerMap: removing container" podUID="830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca" containerName="cleanup"
Oct 30 22:16:08.991326 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991320    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca" containerName="cleanup"
Oct 30 22:16:08.991326 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991329    5087 cpu_manager.go:414] "containerMap: removing container" podUID="1b01d57d-4936-4e45-8b9e-c9ea8e7246fc" containerName="cleanup"
Oct 30 22:16:08.991696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991334    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b01d57d-4936-4e45-8b9e-c9ea8e7246fc" containerName="cleanup"
Oct 30 22:16:08.991696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991342    5087 cpu_manager.go:414] "containerMap: removing container" podUID="5db02748-f530-4034-b739-e9b4b9a263d5" containerName="istio-proxy"
Oct 30 22:16:08.991696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991368    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5db02748-f530-4034-b739-e9b4b9a263d5" containerName="istio-proxy"
Oct 30 22:16:08.991696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991378    5087 cpu_manager.go:414] "containerMap: removing container" podUID="1b48210a-16e2-423e-a233-0d340be66ff5" containerName="cleanup"
Oct 30 22:16:08.991696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991383    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b48210a-16e2-423e-a233-0d340be66ff5" containerName="cleanup"
Oct 30 22:16:08.991696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991399    5087 cpu_manager.go:414] "containerMap: removing container" podUID="830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca" containerName="cleanup"
Oct 30 22:16:08.991696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991403    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca" containerName="cleanup"
Oct 30 22:16:08.991696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991409    5087 cpu_manager.go:414] "containerMap: removing container" podUID="cf713fc3-b05d-453b-8e78-598e256b1e1b" containerName="istio-proxy"
Oct 30 22:16:08.991696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991413    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf713fc3-b05d-453b-8e78-598e256b1e1b" containerName="istio-proxy"
Oct 30 22:16:08.991696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991420    5087 cpu_manager.go:414] "containerMap: removing container" podUID="c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d" containerName="istio-proxy"
Oct 30 22:16:08.991696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991424    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d" containerName="istio-proxy"
Oct 30 22:16:08.991696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991433    5087 cpu_manager.go:414] "containerMap: removing container" podUID="830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca" containerName="cleanup"
Oct 30 22:16:08.991696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991437    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca" containerName="cleanup"
Oct 30 22:16:08.991696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991448    5087 cpu_manager.go:414] "containerMap: removing container" podUID="53f51d9f-f90e-43f9-85de-eab0d05086ee" containerName="istio-proxy"
Oct 30 22:16:08.991696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991452    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53f51d9f-f90e-43f9-85de-eab0d05086ee" containerName="istio-proxy"
Oct 30 22:16:08.991696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991458    5087 cpu_manager.go:414] "containerMap: removing container" podUID="dce092df-6490-433b-a5fa-02df7db0b5a8" containerName="cleanup"
Oct 30 22:16:08.991696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991462    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dce092df-6490-433b-a5fa-02df7db0b5a8" containerName="cleanup"
Oct 30 22:16:08.991696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991528    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf713fc3-b05d-453b-8e78-598e256b1e1b" containerName="istio-proxy"
Oct 30 22:16:08.991696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991535    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9ae56f1-3ae7-40bd-86a4-239cfcbcfe1d" containerName="istio-proxy"
Oct 30 22:16:08.991696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991542    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca" containerName="cleanup"
Oct 30 22:16:08.991696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991548    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b01d57d-4936-4e45-8b9e-c9ea8e7246fc" containerName="cleanup"
Oct 30 22:16:08.991696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991554    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="53f51d9f-f90e-43f9-85de-eab0d05086ee" containerName="istio-proxy"
Oct 30 22:16:08.991696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991559    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="dce092df-6490-433b-a5fa-02df7db0b5a8" containerName="cleanup"
Oct 30 22:16:08.991696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991566    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b48210a-16e2-423e-a233-0d340be66ff5" containerName="cleanup"
Oct 30 22:16:08.991696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991571    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="284b12e8-9e86-4406-8494-360e02159b74" containerName="istio-proxy"
Oct 30 22:16:08.991696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991577    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b48210a-16e2-423e-a233-0d340be66ff5" containerName="cleanup"
Oct 30 22:16:08.991696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991582    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca" containerName="cleanup"
Oct 30 22:16:08.991696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991587    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b01d57d-4936-4e45-8b9e-c9ea8e7246fc" containerName="cleanup"
Oct 30 22:16:08.991696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991592    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="5db02748-f530-4034-b739-e9b4b9a263d5" containerName="istio-proxy"
Oct 30 22:16:08.991696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.991598    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b48210a-16e2-423e-a233-0d340be66ff5" containerName="cleanup"
Oct 30 22:16:08.996183 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:08.996165    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-632149ee-hd66x"
Oct 30 22:16:09.047937 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:09.047901    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:16:09.059690 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:09.059660    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8nvm\" (UniqueName: \"kubernetes.io/projected/f9bc679e-17cc-4b56-908d-093182b7dd78-kube-api-access-s8nvm\") pod \"maas-api-revoke-sub-632149ee-hd66x\" (UID: \"f9bc679e-17cc-4b56-908d-093182b7dd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-632149ee-hd66x"
Oct 30 22:16:09.060136 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:09.060111    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9bc679e-17cc-4b56-908d-093182b7dd78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-632149ee-hd66x\" (UID: \"f9bc679e-17cc-4b56-908d-093182b7dd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-632149ee-hd66x"
Oct 30 22:16:09.060220 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:09.060179    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9bc679e-17cc-4b56-908d-093182b7dd78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-632149ee-hd66x\" (UID: \"f9bc679e-17cc-4b56-908d-093182b7dd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-632149ee-hd66x"
Oct 30 22:16:09.097415 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:09.097388    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:16:09.100089 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:09.100067    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-632149ee-hd66x"]
Oct 30 22:16:09.160922 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:09.160875    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8nvm\" (UniqueName: \"kubernetes.io/projected/f9bc679e-17cc-4b56-908d-093182b7dd78-kube-api-access-s8nvm\") pod \"maas-api-revoke-sub-632149ee-hd66x\" (UID: \"f9bc679e-17cc-4b56-908d-093182b7dd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-632149ee-hd66x"
Oct 30 22:16:09.161115 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:09.160977    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9bc679e-17cc-4b56-908d-093182b7dd78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-632149ee-hd66x\" (UID: \"f9bc679e-17cc-4b56-908d-093182b7dd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-632149ee-hd66x"
Oct 30 22:16:09.161115 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:09.161007    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9bc679e-17cc-4b56-908d-093182b7dd78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-632149ee-hd66x\" (UID: \"f9bc679e-17cc-4b56-908d-093182b7dd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-632149ee-hd66x"
Oct 30 22:16:09.161625 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:09.161606    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9bc679e-17cc-4b56-908d-093182b7dd78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-632149ee-hd66x\" (UID: \"f9bc679e-17cc-4b56-908d-093182b7dd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-632149ee-hd66x"
Oct 30 22:16:09.163466 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:09.163443    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9bc679e-17cc-4b56-908d-093182b7dd78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-632149ee-hd66x\" (UID: \"f9bc679e-17cc-4b56-908d-093182b7dd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-632149ee-hd66x"
Oct 30 22:16:09.217758 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:09.217732    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8nvm\" (UniqueName: \"kubernetes.io/projected/f9bc679e-17cc-4b56-908d-093182b7dd78-kube-api-access-s8nvm\") pod \"maas-api-revoke-sub-632149ee-hd66x\" (UID: \"f9bc679e-17cc-4b56-908d-093182b7dd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-632149ee-hd66x"
Oct 30 22:16:09.309092 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:09.309020    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-632149ee-hd66x"
Oct 30 22:16:09.685038 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:09.685008    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-632149ee-hd66x"]
Oct 30 22:16:09.686152 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:16:09.686123    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9bc679e_17cc_4b56_908d_093182b7dd78.slice/crio-d9fb90d94c89569fd8befe008f3e3562963ccd9afb66cb73f8620a2884b83be6 WatchSource:0}: Error finding container d9fb90d94c89569fd8befe008f3e3562963ccd9afb66cb73f8620a2884b83be6: Status 404 returned error can't find the container with id d9fb90d94c89569fd8befe008f3e3562963ccd9afb66cb73f8620a2884b83be6
Oct 30 22:16:10.526261 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.526239    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 22:16:10.677202 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.677175    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istio-data\") pod \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") "
Oct 30 22:16:10.677421 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.677215    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9f7c360-2a81-4e01-b104-6e9907c6cb39-kube-api-access-9vn2q\" (UniqueName: \"kubernetes.io/projected/b9f7c360-2a81-4e01-b104-6e9907c6cb39-kube-api-access-9vn2q\") pod \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") "
Oct 30 22:16:10.677421 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.677256    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-workload-certs\") pod \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") "
Oct 30 22:16:10.677421 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.677293    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istio-podinfo\") pod \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") "
Oct 30 22:16:10.677421 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.677319    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istio-token\" (UniqueName: \"kubernetes.io/projected/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istio-token\") pod \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") "
Oct 30 22:16:10.677421 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.677378    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-workload-socket\") pod \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") "
Oct 30 22:16:10.677710 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.677504    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-credential-socket\") pod \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") "
Oct 30 22:16:10.677710 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.677530    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-workload-certs" pod "b9f7c360-2a81-4e01-b104-6e9907c6cb39" (UID: "b9f7c360-2a81-4e01-b104-6e9907c6cb39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:10.677710 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.677540    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istio-data" pod "b9f7c360-2a81-4e01-b104-6e9907c6cb39" (UID: "b9f7c360-2a81-4e01-b104-6e9907c6cb39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:10.677710 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.677560    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istiod-ca-cert\") pod \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") "
Oct 30 22:16:10.677710 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.677624    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istio-envoy\") pod \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\" (UID: \"b9f7c360-2a81-4e01-b104-6e9907c6cb39\") "
Oct 30 22:16:10.677986 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.677804    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-workload-socket" pod "b9f7c360-2a81-4e01-b104-6e9907c6cb39" (UID: "b9f7c360-2a81-4e01-b104-6e9907c6cb39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:10.677986 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.677818    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-credential-socket" pod "b9f7c360-2a81-4e01-b104-6e9907c6cb39" (UID: "b9f7c360-2a81-4e01-b104-6e9907c6cb39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:10.677986 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.677885    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istiod-ca-cert" pod "b9f7c360-2a81-4e01-b104-6e9907c6cb39" (UID: "b9f7c360-2a81-4e01-b104-6e9907c6cb39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:10.678315 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.678061    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:10.678315 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.678080    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:10.678315 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.678093    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:10.678315 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.678102    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:10.678315 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.678110    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:10.679831 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.679805    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istio-podinfo" pod "b9f7c360-2a81-4e01-b104-6e9907c6cb39" (UID: "b9f7c360-2a81-4e01-b104-6e9907c6cb39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:16:10.679831 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.679812    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f7c360-2a81-4e01-b104-6e9907c6cb39-kube-api-access-9vn2q" pod "b9f7c360-2a81-4e01-b104-6e9907c6cb39" (UID: "b9f7c360-2a81-4e01-b104-6e9907c6cb39"). InnerVolumeSpecName "kube-api-access-9vn2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:10.679984 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.679841    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istio-token" pod "b9f7c360-2a81-4e01-b104-6e9907c6cb39" (UID: "b9f7c360-2a81-4e01-b104-6e9907c6cb39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:10.680200 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.680186    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istio-envoy" pod "b9f7c360-2a81-4e01-b104-6e9907c6cb39" (UID: "b9f7c360-2a81-4e01-b104-6e9907c6cb39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:10.697165 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.697143    5087 generic.go:356] "Generic (PLEG): container finished" podID="f9bc679e-17cc-4b56-908d-093182b7dd78" containerID="6ace9120c54b06dc4e7b67fbff39dea29c5582d00e974a34f9c12b7d2560a199" exitCode=0
Oct 30 22:16:10.697267 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.697191    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-632149ee-hd66x" event={"ID":"f9bc679e-17cc-4b56-908d-093182b7dd78","Type":"ContainerDied","Data":"6ace9120c54b06dc4e7b67fbff39dea29c5582d00e974a34f9c12b7d2560a199"}
Oct 30 22:16:10.697359 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.697271    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-632149ee-hd66x" event={"ID":"f9bc679e-17cc-4b56-908d-093182b7dd78","Type":"ContainerStarted","Data":"d9fb90d94c89569fd8befe008f3e3562963ccd9afb66cb73f8620a2884b83be6"}
Oct 30 22:16:10.698636 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.698617    5087 generic.go:356] "Generic (PLEG): container finished" podID="b9f7c360-2a81-4e01-b104-6e9907c6cb39" containerID="425a4c700a055308a86b8dd186fbb8ae7514dfc3e3c1c17713e941a2f54c5aaf" exitCode=0
Oct 30 22:16:10.698720 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.698657    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"
Oct 30 22:16:10.698720 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.698675    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l" event={"ID":"b9f7c360-2a81-4e01-b104-6e9907c6cb39","Type":"ContainerDied","Data":"425a4c700a055308a86b8dd186fbb8ae7514dfc3e3c1c17713e941a2f54c5aaf"}
Oct 30 22:16:10.698720 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.698716    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l" event={"ID":"b9f7c360-2a81-4e01-b104-6e9907c6cb39","Type":"ContainerDied","Data":"89d610f4ccf183d28c0be9fba8d67b4d4069171716e7de2d01ee159d8f654b77"}
Oct 30 22:16:10.698826 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.698738    5087 scope.go:122] "RemoveContainer" containerID="425a4c700a055308a86b8dd186fbb8ae7514dfc3e3c1c17713e941a2f54c5aaf"
Oct 30 22:16:10.709040 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.709025    5087 scope.go:122] "RemoveContainer" containerID="425a4c700a055308a86b8dd186fbb8ae7514dfc3e3c1c17713e941a2f54c5aaf"
Oct 30 22:16:10.709321 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:16:10.709303    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"425a4c700a055308a86b8dd186fbb8ae7514dfc3e3c1c17713e941a2f54c5aaf\": container with ID starting with 425a4c700a055308a86b8dd186fbb8ae7514dfc3e3c1c17713e941a2f54c5aaf not found: ID does not exist" containerID="425a4c700a055308a86b8dd186fbb8ae7514dfc3e3c1c17713e941a2f54c5aaf"
Oct 30 22:16:10.709474 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.709328    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"425a4c700a055308a86b8dd186fbb8ae7514dfc3e3c1c17713e941a2f54c5aaf"} err="failed to get container status \"425a4c700a055308a86b8dd186fbb8ae7514dfc3e3c1c17713e941a2f54c5aaf\": rpc error: code = NotFound desc = could not find container \"425a4c700a055308a86b8dd186fbb8ae7514dfc3e3c1c17713e941a2f54c5aaf\": container with ID starting with 425a4c700a055308a86b8dd186fbb8ae7514dfc3e3c1c17713e941a2f54c5aaf not found: ID does not exist"
Oct 30 22:16:10.779428 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.779325    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vn2q\" (UniqueName: \"kubernetes.io/projected/b9f7c360-2a81-4e01-b104-6e9907c6cb39-kube-api-access-9vn2q\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:10.779428 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.779380    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:10.779428 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.779398    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:10.779428 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.779414    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9f7c360-2a81-4e01-b104-6e9907c6cb39-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:10.913429 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.913385    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"]
Oct 30 22:16:10.921705 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:10.921676    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c79cb5-openshift-default-5464c54587-4s94l"]
Oct 30 22:16:11.708475 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:11.708432    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9f7c360-2a81-4e01-b104-6e9907c6cb39" path="/var/lib/kubelet/pods/b9f7c360-2a81-4e01-b104-6e9907c6cb39/volumes"
Oct 30 22:16:11.855613 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:11.855588    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-632149ee-hd66x"
Oct 30 22:16:11.993575 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:11.993531    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9bc679e-17cc-4b56-908d-093182b7dd78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9bc679e-17cc-4b56-908d-093182b7dd78-maas-api-openshift-ca\") pod \"f9bc679e-17cc-4b56-908d-093182b7dd78\" (UID: \"f9bc679e-17cc-4b56-908d-093182b7dd78\") "
Oct 30 22:16:11.993738 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:11.993638    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9bc679e-17cc-4b56-908d-093182b7dd78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9bc679e-17cc-4b56-908d-093182b7dd78-maas-api-xks-ca\") pod \"f9bc679e-17cc-4b56-908d-093182b7dd78\" (UID: \"f9bc679e-17cc-4b56-908d-093182b7dd78\") "
Oct 30 22:16:11.993738 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:11.993728    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9bc679e-17cc-4b56-908d-093182b7dd78-kube-api-access-s8nvm\" (UniqueName: \"kubernetes.io/projected/f9bc679e-17cc-4b56-908d-093182b7dd78-kube-api-access-s8nvm\") pod \"f9bc679e-17cc-4b56-908d-093182b7dd78\" (UID: \"f9bc679e-17cc-4b56-908d-093182b7dd78\") "
Oct 30 22:16:11.993898 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:11.993876    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9bc679e-17cc-4b56-908d-093182b7dd78-maas-api-openshift-ca" pod "f9bc679e-17cc-4b56-908d-093182b7dd78" (UID: "f9bc679e-17cc-4b56-908d-093182b7dd78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:11.994096 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:11.994078    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9bc679e-17cc-4b56-908d-093182b7dd78-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:11.995911 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:11.995882    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9bc679e-17cc-4b56-908d-093182b7dd78-maas-api-xks-ca" pod "f9bc679e-17cc-4b56-908d-093182b7dd78" (UID: "f9bc679e-17cc-4b56-908d-093182b7dd78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:16:11.996359 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:11.996324    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9bc679e-17cc-4b56-908d-093182b7dd78-kube-api-access-s8nvm" pod "f9bc679e-17cc-4b56-908d-093182b7dd78" (UID: "f9bc679e-17cc-4b56-908d-093182b7dd78"). InnerVolumeSpecName "kube-api-access-s8nvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:12.094878 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:12.094837    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8nvm\" (UniqueName: \"kubernetes.io/projected/f9bc679e-17cc-4b56-908d-093182b7dd78-kube-api-access-s8nvm\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:12.094878 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:12.094878    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9bc679e-17cc-4b56-908d-093182b7dd78-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:12.713381 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:12.713333    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-632149ee-hd66x"
Oct 30 22:16:12.713381 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:12.713371    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-632149ee-hd66x" event={"ID":"f9bc679e-17cc-4b56-908d-093182b7dd78","Type":"ContainerDied","Data":"d9fb90d94c89569fd8befe008f3e3562963ccd9afb66cb73f8620a2884b83be6"}
Oct 30 22:16:12.713833 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:12.713417    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d9fb90d94c89569fd8befe008f3e3562963ccd9afb66cb73f8620a2884b83be6"
Oct 30 22:16:14.044992 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:14.044958    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dde8a228-xzmw8"]
Oct 30 22:16:14.094413 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:14.094376    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dde8a228-xzmw8"]
Oct 30 22:16:15.709287 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:15.709248    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cde2a2e-3765-4cd3-a1a8-ac979f272afc" path="/var/lib/kubelet/pods/8cde2a2e-3765-4cd3-a1a8-ac979f272afc/volumes"
Oct 30 22:16:17.168318 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:16:17.168279    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:16:20.188463 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:20.188426    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-80c366-wz6cs"]
Oct 30 22:16:20.188929 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:20.188914    5087 cpu_manager.go:414] "containerMap: removing container" podUID="1b01d57d-4936-4e45-8b9e-c9ea8e7246fc" containerName="cleanup"
Oct 30 22:16:20.189020 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:20.188930    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b01d57d-4936-4e45-8b9e-c9ea8e7246fc" containerName="cleanup"
Oct 30 22:16:20.189020 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:20.188943    5087 cpu_manager.go:414] "containerMap: removing container" podUID="f9bc679e-17cc-4b56-908d-093182b7dd78" containerName="revoke-keys"
Oct 30 22:16:20.189020 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:20.188948    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9bc679e-17cc-4b56-908d-093182b7dd78" containerName="revoke-keys"
Oct 30 22:16:20.189020 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:20.188968    5087 cpu_manager.go:414] "containerMap: removing container" podUID="1b48210a-16e2-423e-a233-0d340be66ff5" containerName="cleanup"
Oct 30 22:16:20.189020 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:20.188973    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b48210a-16e2-423e-a233-0d340be66ff5" containerName="cleanup"
Oct 30 22:16:20.189020 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:20.188993    5087 cpu_manager.go:414] "containerMap: removing container" podUID="b9f7c360-2a81-4e01-b104-6e9907c6cb39" containerName="istio-proxy"
Oct 30 22:16:20.189020 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:20.188998    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9f7c360-2a81-4e01-b104-6e9907c6cb39" containerName="istio-proxy"
Oct 30 22:16:20.189693 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:20.189086    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9bc679e-17cc-4b56-908d-093182b7dd78" containerName="revoke-keys"
Oct 30 22:16:20.189693 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:20.189095    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b01d57d-4936-4e45-8b9e-c9ea8e7246fc" containerName="cleanup"
Oct 30 22:16:20.189693 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:20.189105    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9f7c360-2a81-4e01-b104-6e9907c6cb39" containerName="istio-proxy"
Oct 30 22:16:20.189693 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:20.189111    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="830b2eaa-a3e3-469d-ad75-96f4d2a3a2ca" containerName="cleanup"
Oct 30 22:16:20.192276 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:20.192260    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-80c366-wz6cs"
Oct 30 22:16:20.249670 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:20.249638    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-80c366-wz6cs"]
Oct 30 22:16:20.281354 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:20.281308    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ee2e12-e863-4853-9e41-1e6571cfed6b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-80c366-wz6cs\" (UID: \"45ee2e12-e863-4853-9e41-1e6571cfed6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-80c366-wz6cs"
Oct 30 22:16:20.281568 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:20.281542    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb9zt\" (UniqueName: \"kubernetes.io/projected/45ee2e12-e863-4853-9e41-1e6571cfed6b-kube-api-access-mb9zt\") pod \"maas-api-revoke-keys-e2e-worker-w1-80c366-wz6cs\" (UID: \"45ee2e12-e863-4853-9e41-1e6571cfed6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-80c366-wz6cs"
Oct 30 22:16:20.281680 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:20.281660    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ee2e12-e863-4853-9e41-1e6571cfed6b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-80c366-wz6cs\" (UID: \"45ee2e12-e863-4853-9e41-1e6571cfed6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-80c366-wz6cs"
Oct 30 22:16:20.295652 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:20.295626    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:16:20.295805 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:20.295717    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:16:20.383249 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:20.383219    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb9zt\" (UniqueName: \"kubernetes.io/projected/45ee2e12-e863-4853-9e41-1e6571cfed6b-kube-api-access-mb9zt\") pod \"maas-api-revoke-keys-e2e-worker-w1-80c366-wz6cs\" (UID: \"45ee2e12-e863-4853-9e41-1e6571cfed6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-80c366-wz6cs"
Oct 30 22:16:20.383490 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:20.383299    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ee2e12-e863-4853-9e41-1e6571cfed6b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-80c366-wz6cs\" (UID: \"45ee2e12-e863-4853-9e41-1e6571cfed6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-80c366-wz6cs"
Oct 30 22:16:20.383490 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:20.383390    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ee2e12-e863-4853-9e41-1e6571cfed6b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-80c366-wz6cs\" (UID: \"45ee2e12-e863-4853-9e41-1e6571cfed6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-80c366-wz6cs"
Oct 30 22:16:20.384083 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:20.384062    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ee2e12-e863-4853-9e41-1e6571cfed6b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-80c366-wz6cs\" (UID: \"45ee2e12-e863-4853-9e41-1e6571cfed6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-80c366-wz6cs"
Oct 30 22:16:20.385970 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:20.385943    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ee2e12-e863-4853-9e41-1e6571cfed6b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-80c366-wz6cs\" (UID: \"45ee2e12-e863-4853-9e41-1e6571cfed6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-80c366-wz6cs"
Oct 30 22:16:20.440490 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:20.440408    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb9zt\" (UniqueName: \"kubernetes.io/projected/45ee2e12-e863-4853-9e41-1e6571cfed6b-kube-api-access-mb9zt\") pod \"maas-api-revoke-keys-e2e-worker-w1-80c366-wz6cs\" (UID: \"45ee2e12-e863-4853-9e41-1e6571cfed6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-80c366-wz6cs"
Oct 30 22:16:20.505604 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:20.505568    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-80c366-wz6cs"
Oct 30 22:16:20.881110 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:20.881076    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-80c366-wz6cs"]
Oct 30 22:16:20.881412 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:16:20.881387    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ee2e12_e863_4853_9e41_1e6571cfed6b.slice/crio-3b45646ce21ed25d035856de6b07eac37b90bf33f134724000aad28d52706e75 WatchSource:0}: Error finding container 3b45646ce21ed25d035856de6b07eac37b90bf33f134724000aad28d52706e75: Status 404 returned error can't find the container with id 3b45646ce21ed25d035856de6b07eac37b90bf33f134724000aad28d52706e75
Oct 30 22:16:21.776169 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:21.776142    5087 generic.go:356] "Generic (PLEG): container finished" podID="45ee2e12-e863-4853-9e41-1e6571cfed6b" containerID="f91d81855d20539814259143f047d9ffe76821db59e09ddd3d5093a255071487" exitCode=0
Oct 30 22:16:21.776524 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:21.776212    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-80c366-wz6cs" event={"ID":"45ee2e12-e863-4853-9e41-1e6571cfed6b","Type":"ContainerDied","Data":"f91d81855d20539814259143f047d9ffe76821db59e09ddd3d5093a255071487"}
Oct 30 22:16:21.776524 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:21.776253    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-80c366-wz6cs" event={"ID":"45ee2e12-e863-4853-9e41-1e6571cfed6b","Type":"ContainerStarted","Data":"3b45646ce21ed25d035856de6b07eac37b90bf33f134724000aad28d52706e75"}
Oct 30 22:16:22.914582 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:22.914559    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-80c366-wz6cs"
Oct 30 22:16:23.008846 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:23.008809    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45ee2e12-e863-4853-9e41-1e6571cfed6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ee2e12-e863-4853-9e41-1e6571cfed6b-maas-api-openshift-ca\") pod \"45ee2e12-e863-4853-9e41-1e6571cfed6b\" (UID: \"45ee2e12-e863-4853-9e41-1e6571cfed6b\") "
Oct 30 22:16:23.009081 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:23.008885    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45ee2e12-e863-4853-9e41-1e6571cfed6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ee2e12-e863-4853-9e41-1e6571cfed6b-maas-api-xks-ca\") pod \"45ee2e12-e863-4853-9e41-1e6571cfed6b\" (UID: \"45ee2e12-e863-4853-9e41-1e6571cfed6b\") "
Oct 30 22:16:23.009081 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:23.008956    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45ee2e12-e863-4853-9e41-1e6571cfed6b-kube-api-access-mb9zt\" (UniqueName: \"kubernetes.io/projected/45ee2e12-e863-4853-9e41-1e6571cfed6b-kube-api-access-mb9zt\") pod \"45ee2e12-e863-4853-9e41-1e6571cfed6b\" (UID: \"45ee2e12-e863-4853-9e41-1e6571cfed6b\") "
Oct 30 22:16:23.009257 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:23.009228    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ee2e12-e863-4853-9e41-1e6571cfed6b-maas-api-openshift-ca" pod "45ee2e12-e863-4853-9e41-1e6571cfed6b" (UID: "45ee2e12-e863-4853-9e41-1e6571cfed6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:23.011694 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:23.011666    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45ee2e12-e863-4853-9e41-1e6571cfed6b-maas-api-xks-ca" pod "45ee2e12-e863-4853-9e41-1e6571cfed6b" (UID: "45ee2e12-e863-4853-9e41-1e6571cfed6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:16:23.011811 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:23.011753    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ee2e12-e863-4853-9e41-1e6571cfed6b-kube-api-access-mb9zt" pod "45ee2e12-e863-4853-9e41-1e6571cfed6b" (UID: "45ee2e12-e863-4853-9e41-1e6571cfed6b"). InnerVolumeSpecName "kube-api-access-mb9zt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:23.110683 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:23.110603    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45ee2e12-e863-4853-9e41-1e6571cfed6b-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:23.110683 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:23.110632    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45ee2e12-e863-4853-9e41-1e6571cfed6b-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:23.110683 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:23.110642    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb9zt\" (UniqueName: \"kubernetes.io/projected/45ee2e12-e863-4853-9e41-1e6571cfed6b-kube-api-access-mb9zt\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:23.790052 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:23.790025    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-80c366-wz6cs"
Oct 30 22:16:23.790052 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:23.790042    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-80c366-wz6cs" event={"ID":"45ee2e12-e863-4853-9e41-1e6571cfed6b","Type":"ContainerDied","Data":"3b45646ce21ed25d035856de6b07eac37b90bf33f134724000aad28d52706e75"}
Oct 30 22:16:23.790273 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:23.790095    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b45646ce21ed25d035856de6b07eac37b90bf33f134724000aad28d52706e75"
Oct 30 22:16:24.657894 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:24.657854    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"]
Oct 30 22:16:24.707520 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:24.707480    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"]
Oct 30 22:16:29.633956 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:29.633926    5087 scope.go:122] "RemoveContainer" containerID="4df97a8191b4920276f9b54cb1ece64226631132e0a031690ec0189def5bcae8"
Oct 30 22:16:29.645967 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:29.645936    5087 scope.go:122] "RemoveContainer" containerID="1ec58bd1acd788ba0bc9998c85c6e072aa1a20b46744e9f44b97c3bfc7c0a727"
Oct 30 22:16:29.657126 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:29.657104    5087 scope.go:122] "RemoveContainer" containerID="3dff0e4cbdba5f4ad27ebb3677cb7d5fadc58147d7cf0b3bc5c0005ac9e98dff"
Oct 30 22:16:29.666971 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:29.666944    5087 scope.go:122] "RemoveContainer" containerID="6fb84d79935898195cc4a743e0d535fb7ab81832e6d3e9866d76d40f261a504c"
Oct 30 22:16:29.676826 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:29.676804    5087 scope.go:122] "RemoveContainer" containerID="7961b3c67f4a9b559de92d0df2455b636c15eb6717dbbe2c1e7e7672fea12668"
Oct 30 22:16:29.685899 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:29.685872    5087 scope.go:122] "RemoveContainer" containerID="69b9a962ae44dd29444fac58a57b945627f3fadba7cf4af56b15fc767731fc6b"
Oct 30 22:16:39.678283 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:39.678219    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml" podUID="01fdf382-6833-463a-bed9-c7c096166819" containerName="main" containerID="cri-o://da74c85e4036800e1d6599fb25537dfe99fff96d90c7e5a51504086cb975fb57" gracePeriod=45
Oct 30 22:16:39.731090 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:39.731047    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main" containerID="cri-o://8d2988189c44731c4b36c5f22dd19b76e057a34260654fa670b473456dd838a4" gracePeriod=45
Oct 30 22:16:39.905386 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:39.903054    5087 generic.go:356] "Generic (PLEG): container finished" podID="01fdf382-6833-463a-bed9-c7c096166819" containerID="da74c85e4036800e1d6599fb25537dfe99fff96d90c7e5a51504086cb975fb57" exitCode=0
Oct 30 22:16:39.905386 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:39.903250    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml" event={"ID":"01fdf382-6833-463a-bed9-c7c096166819","Type":"ContainerDied","Data":"da74c85e4036800e1d6599fb25537dfe99fff96d90c7e5a51504086cb975fb57"}
Oct 30 22:16:39.905630 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:39.905596    5087 generic.go:356] "Generic (PLEG): container finished" podID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerID="8d2988189c44731c4b36c5f22dd19b76e057a34260654fa670b473456dd838a4" exitCode=0
Oct 30 22:16:39.905728 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:39.905710    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" event={"ID":"ca9b0116-60ed-4471-aba6-a2cc23eb8480","Type":"ContainerDied","Data":"8d2988189c44731c4b36c5f22dd19b76e057a34260654fa670b473456dd838a4"}
Oct 30 22:16:39.990091 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:39.990067    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 22:16:40.033148 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.033125    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 22:16:40.076091 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.076063    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca9b0116-60ed-4471-aba6-a2cc23eb8480-home\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0116-60ed-4471-aba6-a2cc23eb8480-home\") pod \"ca9b0116-60ed-4471-aba6-a2cc23eb8480\" (UID: \"ca9b0116-60ed-4471-aba6-a2cc23eb8480\") "
Oct 30 22:16:40.076230 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.076104    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca9b0116-60ed-4471-aba6-a2cc23eb8480-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca9b0116-60ed-4471-aba6-a2cc23eb8480-tls-certs\") pod \"ca9b0116-60ed-4471-aba6-a2cc23eb8480\" (UID: \"ca9b0116-60ed-4471-aba6-a2cc23eb8480\") "
Oct 30 22:16:40.076230 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.076140    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01fdf382-6833-463a-bed9-c7c096166819-dshm\" (UniqueName: \"kubernetes.io/empty-dir/01fdf382-6833-463a-bed9-c7c096166819-dshm\") pod \"01fdf382-6833-463a-bed9-c7c096166819\" (UID: \"01fdf382-6833-463a-bed9-c7c096166819\") "
Oct 30 22:16:40.076230 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.076187    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01fdf382-6833-463a-bed9-c7c096166819-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01fdf382-6833-463a-bed9-c7c096166819-tmp-dir\") pod \"01fdf382-6833-463a-bed9-c7c096166819\" (UID: \"01fdf382-6833-463a-bed9-c7c096166819\") "
Oct 30 22:16:40.076230 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.076214    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01fdf382-6833-463a-bed9-c7c096166819-tls-certs\" (UniqueName: \"kubernetes.io/secret/01fdf382-6833-463a-bed9-c7c096166819-tls-certs\") pod \"01fdf382-6833-463a-bed9-c7c096166819\" (UID: \"01fdf382-6833-463a-bed9-c7c096166819\") "
Oct 30 22:16:40.076498 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.076245    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca9b0116-60ed-4471-aba6-a2cc23eb8480-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0116-60ed-4471-aba6-a2cc23eb8480-tmp-dir\") pod \"ca9b0116-60ed-4471-aba6-a2cc23eb8480\" (UID: \"ca9b0116-60ed-4471-aba6-a2cc23eb8480\") "
Oct 30 22:16:40.076498 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.076315    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01fdf382-6833-463a-bed9-c7c096166819-kube-api-access-j7tnc\" (UniqueName: \"kubernetes.io/projected/01fdf382-6833-463a-bed9-c7c096166819-kube-api-access-j7tnc\") pod \"01fdf382-6833-463a-bed9-c7c096166819\" (UID: \"01fdf382-6833-463a-bed9-c7c096166819\") "
Oct 30 22:16:40.076498 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.076380    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca9b0116-60ed-4471-aba6-a2cc23eb8480-home" pod "ca9b0116-60ed-4471-aba6-a2cc23eb8480" (UID: "ca9b0116-60ed-4471-aba6-a2cc23eb8480"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:40.076498 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.076399    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca9b0116-60ed-4471-aba6-a2cc23eb8480-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0116-60ed-4471-aba6-a2cc23eb8480-model-cache\") pod \"ca9b0116-60ed-4471-aba6-a2cc23eb8480\" (UID: \"ca9b0116-60ed-4471-aba6-a2cc23eb8480\") "
Oct 30 22:16:40.076498 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.076468    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01fdf382-6833-463a-bed9-c7c096166819-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01fdf382-6833-463a-bed9-c7c096166819-model-cache\") pod \"01fdf382-6833-463a-bed9-c7c096166819\" (UID: \"01fdf382-6833-463a-bed9-c7c096166819\") "
Oct 30 22:16:40.076788 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.076519    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01fdf382-6833-463a-bed9-c7c096166819-tmp-dir" pod "01fdf382-6833-463a-bed9-c7c096166819" (UID: "01fdf382-6833-463a-bed9-c7c096166819"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:40.076788 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.076532    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca9b0116-60ed-4471-aba6-a2cc23eb8480-kube-api-access-tdlmc\" (UniqueName: \"kubernetes.io/projected/ca9b0116-60ed-4471-aba6-a2cc23eb8480-kube-api-access-tdlmc\") pod \"ca9b0116-60ed-4471-aba6-a2cc23eb8480\" (UID: \"ca9b0116-60ed-4471-aba6-a2cc23eb8480\") "
Oct 30 22:16:40.076788 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.076568    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca9b0116-60ed-4471-aba6-a2cc23eb8480-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0116-60ed-4471-aba6-a2cc23eb8480-dshm\") pod \"ca9b0116-60ed-4471-aba6-a2cc23eb8480\" (UID: \"ca9b0116-60ed-4471-aba6-a2cc23eb8480\") "
Oct 30 22:16:40.076788 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.076614    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01fdf382-6833-463a-bed9-c7c096166819-home\" (UniqueName: \"kubernetes.io/empty-dir/01fdf382-6833-463a-bed9-c7c096166819-home\") pod \"01fdf382-6833-463a-bed9-c7c096166819\" (UID: \"01fdf382-6833-463a-bed9-c7c096166819\") "
Oct 30 22:16:40.076788 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.076673    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca9b0116-60ed-4471-aba6-a2cc23eb8480-model-cache" pod "ca9b0116-60ed-4471-aba6-a2cc23eb8480" (UID: "ca9b0116-60ed-4471-aba6-a2cc23eb8480"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:40.076788 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.076750    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01fdf382-6833-463a-bed9-c7c096166819-model-cache" pod "01fdf382-6833-463a-bed9-c7c096166819" (UID: "01fdf382-6833-463a-bed9-c7c096166819"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:40.076788 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.076757    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca9b0116-60ed-4471-aba6-a2cc23eb8480-tmp-dir" pod "ca9b0116-60ed-4471-aba6-a2cc23eb8480" (UID: "ca9b0116-60ed-4471-aba6-a2cc23eb8480"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:40.077361 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.077053    5087 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01fdf382-6833-463a-bed9-c7c096166819-tmp-dir\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:40.077361 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.077207    5087 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0116-60ed-4471-aba6-a2cc23eb8480-tmp-dir\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:40.077361 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.077223    5087 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0116-60ed-4471-aba6-a2cc23eb8480-model-cache\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:40.077361 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.077236    5087 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01fdf382-6833-463a-bed9-c7c096166819-model-cache\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:40.077361 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.077251    5087 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0116-60ed-4471-aba6-a2cc23eb8480-home\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:40.077647 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.077370    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01fdf382-6833-463a-bed9-c7c096166819-home" pod "01fdf382-6833-463a-bed9-c7c096166819" (UID: "01fdf382-6833-463a-bed9-c7c096166819"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:40.078781 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.078754    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01fdf382-6833-463a-bed9-c7c096166819-dshm" pod "01fdf382-6833-463a-bed9-c7c096166819" (UID: "01fdf382-6833-463a-bed9-c7c096166819"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:40.078889 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.078790    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca9b0116-60ed-4471-aba6-a2cc23eb8480-tls-certs" pod "ca9b0116-60ed-4471-aba6-a2cc23eb8480" (UID: "ca9b0116-60ed-4471-aba6-a2cc23eb8480"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:16:40.079061 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.079040    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01fdf382-6833-463a-bed9-c7c096166819-tls-certs" pod "01fdf382-6833-463a-bed9-c7c096166819" (UID: "01fdf382-6833-463a-bed9-c7c096166819"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:16:40.079173 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.079156    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01fdf382-6833-463a-bed9-c7c096166819-kube-api-access-j7tnc" pod "01fdf382-6833-463a-bed9-c7c096166819" (UID: "01fdf382-6833-463a-bed9-c7c096166819"). InnerVolumeSpecName "kube-api-access-j7tnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:40.079240 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.079219    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca9b0116-60ed-4471-aba6-a2cc23eb8480-dshm" pod "ca9b0116-60ed-4471-aba6-a2cc23eb8480" (UID: "ca9b0116-60ed-4471-aba6-a2cc23eb8480"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:16:40.079407 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.079391    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca9b0116-60ed-4471-aba6-a2cc23eb8480-kube-api-access-tdlmc" pod "ca9b0116-60ed-4471-aba6-a2cc23eb8480" (UID: "ca9b0116-60ed-4471-aba6-a2cc23eb8480"). InnerVolumeSpecName "kube-api-access-tdlmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:40.178811 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.178774    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdlmc\" (UniqueName: \"kubernetes.io/projected/ca9b0116-60ed-4471-aba6-a2cc23eb8480-kube-api-access-tdlmc\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:40.178978 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.178815    5087 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca9b0116-60ed-4471-aba6-a2cc23eb8480-dshm\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:40.178978 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.178831    5087 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01fdf382-6833-463a-bed9-c7c096166819-home\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:40.178978 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.178844    5087 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca9b0116-60ed-4471-aba6-a2cc23eb8480-tls-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:40.178978 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.178859    5087 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01fdf382-6833-463a-bed9-c7c096166819-dshm\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:40.178978 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.178872    5087 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01fdf382-6833-463a-bed9-c7c096166819-tls-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:40.178978 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.178886    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7tnc\" (UniqueName: \"kubernetes.io/projected/01fdf382-6833-463a-bed9-c7c096166819-kube-api-access-j7tnc\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:40.913849 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.913820    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"
Oct 30 22:16:40.913849 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.913829    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml" event={"ID":"01fdf382-6833-463a-bed9-c7c096166819","Type":"ContainerDied","Data":"1b91b9ac68a4641e1ee64933c30c3dfa8fe9b34ef472502ff2045329cc4f50a0"}
Oct 30 22:16:40.914484 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.913883    5087 scope.go:122] "RemoveContainer" containerID="da74c85e4036800e1d6599fb25537dfe99fff96d90c7e5a51504086cb975fb57"
Oct 30 22:16:40.915825 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.915802    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl" event={"ID":"ca9b0116-60ed-4471-aba6-a2cc23eb8480","Type":"ContainerDied","Data":"cf8a132c6f8f63e067c9a2ff2641e33b7d6738e36eb7d8ced103154aa318e7c1"}
Oct 30 22:16:40.916002 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.915857    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"
Oct 30 22:16:40.924729 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:40.924711    5087 scope.go:122] "RemoveContainer" containerID="8d2988189c44731c4b36c5f22dd19b76e057a34260654fa670b473456dd838a4"
Oct 30 22:16:41.075677 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:41.075640    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"]
Oct 30 22:16:41.125437 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:41.125403    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-80c366/simulator-w1-80c366-kserve-5f67bc7b86-f9rsl"]
Oct 30 22:16:41.233244 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:41.233207    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"]
Oct 30 22:16:41.235580 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:41.235559    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-80c366/distinct-2-w1-80c366-kserve-58f958b885-rvsml"]
Oct 30 22:16:41.708039 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:41.708002    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01fdf382-6833-463a-bed9-c7c096166819" path="/var/lib/kubelet/pods/01fdf382-6833-463a-bed9-c7c096166819/volumes"
Oct 30 22:16:41.708531 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:41.708509    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" path="/var/lib/kubelet/pods/ca9b0116-60ed-4471-aba6-a2cc23eb8480/volumes"
Oct 30 22:16:46.192059 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:46.192025    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52bb6f1e-ndzqn"]
Oct 30 22:16:46.192769 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:46.192748    5087 cpu_manager.go:414] "containerMap: removing container" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main"
Oct 30 22:16:46.192880 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:46.192771    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main"
Oct 30 22:16:46.192880 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:46.192785    5087 cpu_manager.go:414] "containerMap: removing container" podUID="01fdf382-6833-463a-bed9-c7c096166819" containerName="main"
Oct 30 22:16:46.192880 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:46.192793    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01fdf382-6833-463a-bed9-c7c096166819" containerName="main"
Oct 30 22:16:46.192880 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:46.192823    5087 cpu_manager.go:414] "containerMap: removing container" podUID="45ee2e12-e863-4853-9e41-1e6571cfed6b" containerName="revoke-keys"
Oct 30 22:16:46.192880 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:46.192831    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45ee2e12-e863-4853-9e41-1e6571cfed6b" containerName="revoke-keys"
Oct 30 22:16:46.193143 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:46.192943    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca9b0116-60ed-4471-aba6-a2cc23eb8480" containerName="main"
Oct 30 22:16:46.193143 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:46.192956    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="01fdf382-6833-463a-bed9-c7c096166819" containerName="main"
Oct 30 22:16:46.193143 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:46.192971    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="45ee2e12-e863-4853-9e41-1e6571cfed6b" containerName="revoke-keys"
Oct 30 22:16:46.197796 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:46.197776    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52bb6f1e-ndzqn"
Oct 30 22:16:46.240396 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:46.240366    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt525\" (UniqueName: \"kubernetes.io/projected/ace33c00-1c8a-4626-a72a-4d6e74a86a16-kube-api-access-jt525\") pod \"maas-api-revoke-sub-52bb6f1e-ndzqn\" (UID: \"ace33c00-1c8a-4626-a72a-4d6e74a86a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52bb6f1e-ndzqn"
Oct 30 22:16:46.240543 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:46.240412    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ace33c00-1c8a-4626-a72a-4d6e74a86a16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52bb6f1e-ndzqn\" (UID: \"ace33c00-1c8a-4626-a72a-4d6e74a86a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52bb6f1e-ndzqn"
Oct 30 22:16:46.240606 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:46.240543    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ace33c00-1c8a-4626-a72a-4d6e74a86a16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52bb6f1e-ndzqn\" (UID: \"ace33c00-1c8a-4626-a72a-4d6e74a86a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52bb6f1e-ndzqn"
Oct 30 22:16:46.255492 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:46.255465    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52bb6f1e-ndzqn"]
Oct 30 22:16:46.301545 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:46.301517    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:16:46.301545 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:46.301524    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:16:46.341282 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:46.341253    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt525\" (UniqueName: \"kubernetes.io/projected/ace33c00-1c8a-4626-a72a-4d6e74a86a16-kube-api-access-jt525\") pod \"maas-api-revoke-sub-52bb6f1e-ndzqn\" (UID: \"ace33c00-1c8a-4626-a72a-4d6e74a86a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52bb6f1e-ndzqn"
Oct 30 22:16:46.341473 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:46.341286    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ace33c00-1c8a-4626-a72a-4d6e74a86a16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52bb6f1e-ndzqn\" (UID: \"ace33c00-1c8a-4626-a72a-4d6e74a86a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52bb6f1e-ndzqn"
Oct 30 22:16:46.341473 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:46.341410    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ace33c00-1c8a-4626-a72a-4d6e74a86a16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52bb6f1e-ndzqn\" (UID: \"ace33c00-1c8a-4626-a72a-4d6e74a86a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52bb6f1e-ndzqn"
Oct 30 22:16:46.341963 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:46.341945    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ace33c00-1c8a-4626-a72a-4d6e74a86a16-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52bb6f1e-ndzqn\" (UID: \"ace33c00-1c8a-4626-a72a-4d6e74a86a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52bb6f1e-ndzqn"
Oct 30 22:16:46.343892 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:46.343866    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ace33c00-1c8a-4626-a72a-4d6e74a86a16-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52bb6f1e-ndzqn\" (UID: \"ace33c00-1c8a-4626-a72a-4d6e74a86a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52bb6f1e-ndzqn"
Oct 30 22:16:46.399239 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:46.399209    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt525\" (UniqueName: \"kubernetes.io/projected/ace33c00-1c8a-4626-a72a-4d6e74a86a16-kube-api-access-jt525\") pod \"maas-api-revoke-sub-52bb6f1e-ndzqn\" (UID: \"ace33c00-1c8a-4626-a72a-4d6e74a86a16\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52bb6f1e-ndzqn"
Oct 30 22:16:46.511554 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:46.511522    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52bb6f1e-ndzqn"
Oct 30 22:16:46.894888 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:46.894862    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52bb6f1e-ndzqn"]
Oct 30 22:16:46.897038 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:16:46.897009    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace33c00_1c8a_4626_a72a_4d6e74a86a16.slice/crio-ad0541809b1e4d2232f81bc7eff1a3ca2cb004b07b91caae569f46d804a8d327 WatchSource:0}: Error finding container ad0541809b1e4d2232f81bc7eff1a3ca2cb004b07b91caae569f46d804a8d327: Status 404 returned error can't find the container with id ad0541809b1e4d2232f81bc7eff1a3ca2cb004b07b91caae569f46d804a8d327
Oct 30 22:16:46.953437 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:46.953409    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52bb6f1e-ndzqn" event={"ID":"ace33c00-1c8a-4626-a72a-4d6e74a86a16","Type":"ContainerStarted","Data":"ad0541809b1e4d2232f81bc7eff1a3ca2cb004b07b91caae569f46d804a8d327"}
Oct 30 22:16:47.168842 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:16:47.168783    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:16:47.961310 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:47.961225    5087 generic.go:356] "Generic (PLEG): container finished" podID="ace33c00-1c8a-4626-a72a-4d6e74a86a16" containerID="a57e1f5dc2daf67b15ce94d642f109ad1ad5a123438ec2bacdc42d12bf35f94c" exitCode=0
Oct 30 22:16:47.961310 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:47.961273    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52bb6f1e-ndzqn" event={"ID":"ace33c00-1c8a-4626-a72a-4d6e74a86a16","Type":"ContainerDied","Data":"a57e1f5dc2daf67b15ce94d642f109ad1ad5a123438ec2bacdc42d12bf35f94c"}
Oct 30 22:16:49.113052 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:49.113024    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52bb6f1e-ndzqn"
Oct 30 22:16:49.172174 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:49.172148    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ace33c00-1c8a-4626-a72a-4d6e74a86a16-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ace33c00-1c8a-4626-a72a-4d6e74a86a16-maas-api-openshift-ca\") pod \"ace33c00-1c8a-4626-a72a-4d6e74a86a16\" (UID: \"ace33c00-1c8a-4626-a72a-4d6e74a86a16\") "
Oct 30 22:16:49.172336 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:49.172295    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ace33c00-1c8a-4626-a72a-4d6e74a86a16-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ace33c00-1c8a-4626-a72a-4d6e74a86a16-maas-api-xks-ca\") pod \"ace33c00-1c8a-4626-a72a-4d6e74a86a16\" (UID: \"ace33c00-1c8a-4626-a72a-4d6e74a86a16\") "
Oct 30 22:16:49.172336 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:49.172328    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ace33c00-1c8a-4626-a72a-4d6e74a86a16-kube-api-access-jt525\" (UniqueName: \"kubernetes.io/projected/ace33c00-1c8a-4626-a72a-4d6e74a86a16-kube-api-access-jt525\") pod \"ace33c00-1c8a-4626-a72a-4d6e74a86a16\" (UID: \"ace33c00-1c8a-4626-a72a-4d6e74a86a16\") "
Oct 30 22:16:49.172604 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:49.172567    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ace33c00-1c8a-4626-a72a-4d6e74a86a16-maas-api-openshift-ca" pod "ace33c00-1c8a-4626-a72a-4d6e74a86a16" (UID: "ace33c00-1c8a-4626-a72a-4d6e74a86a16"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:49.172790 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:49.172772    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ace33c00-1c8a-4626-a72a-4d6e74a86a16-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:49.174561 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:49.174536    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace33c00-1c8a-4626-a72a-4d6e74a86a16-kube-api-access-jt525" pod "ace33c00-1c8a-4626-a72a-4d6e74a86a16" (UID: "ace33c00-1c8a-4626-a72a-4d6e74a86a16"). InnerVolumeSpecName "kube-api-access-jt525". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:49.174661 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:49.174575    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ace33c00-1c8a-4626-a72a-4d6e74a86a16-maas-api-xks-ca" pod "ace33c00-1c8a-4626-a72a-4d6e74a86a16" (UID: "ace33c00-1c8a-4626-a72a-4d6e74a86a16"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:16:49.273686 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:49.273647    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ace33c00-1c8a-4626-a72a-4d6e74a86a16-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:49.273686 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:49.273674    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt525\" (UniqueName: \"kubernetes.io/projected/ace33c00-1c8a-4626-a72a-4d6e74a86a16-kube-api-access-jt525\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:16:49.975293 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:49.975219    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52bb6f1e-ndzqn"
Oct 30 22:16:49.975293 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:49.975235    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52bb6f1e-ndzqn" event={"ID":"ace33c00-1c8a-4626-a72a-4d6e74a86a16","Type":"ContainerDied","Data":"ad0541809b1e4d2232f81bc7eff1a3ca2cb004b07b91caae569f46d804a8d327"}
Oct 30 22:16:49.975293 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:49.975284    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad0541809b1e4d2232f81bc7eff1a3ca2cb004b07b91caae569f46d804a8d327"
Oct 30 22:16:50.081621 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:50.081580    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82c035ad-wl8nc"]
Oct 30 22:16:50.083902 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:50.083877    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f87c2fdb-jrs24"]
Oct 30 22:16:50.085856 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:50.085835    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82c035ad-wl8nc"]
Oct 30 22:16:50.104485 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:50.104447    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f87c2fdb-jrs24"]
Oct 30 22:16:51.709972 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:51.709938    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36ffceb8-7a93-46f7-897d-4a3a72b2864f" path="/var/lib/kubelet/pods/36ffceb8-7a93-46f7-897d-4a3a72b2864f/volumes"
Oct 30 22:16:51.710395 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:16:51.710333    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40934d2d-712a-4070-935e-cbeb5120c309" path="/var/lib/kubelet/pods/40934d2d-712a-4070-935e-cbeb5120c309/volumes"
Oct 30 22:17:17.168282 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:17:17.168247    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:17:22.235048 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:17:22.235016    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-80c366-wz6cs"]
Oct 30 22:17:22.284232 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:17:22.284194    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-80c366-wz6cs"]
Oct 30 22:17:23.707777 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:17:23.707745    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45ee2e12-e863-4853-9e41-1e6571cfed6b" path="/var/lib/kubelet/pods/45ee2e12-e863-4853-9e41-1e6571cfed6b/volumes"
Oct 30 22:17:29.773762 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:17:29.773730    5087 scope.go:122] "RemoveContainer" containerID="f31f000eb28fa6f8c76a10e4b8cc57c12a744ae8e5d717ec5fb1490301846e12"
Oct 30 22:17:29.783766 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:17:29.783744    5087 scope.go:122] "RemoveContainer" containerID="70bc5dd5d10d5c0bef0dc603aa17dc5ba95f58cf93ad2ec848b7082b73258cbd"
Oct 30 22:17:29.792630 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:17:29.792613    5087 scope.go:122] "RemoveContainer" containerID="f461d9615c419d3c16320c9848c9ad1048833e923609186944f3bfc146af8557"
Oct 30 22:17:31.055529 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:17:31.055491    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af27e8d0-zrk77"]
Oct 30 22:17:31.104650 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:17:31.104614    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af27e8d0-zrk77"]
Oct 30 22:17:31.708084 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:17:31.708054    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96a18860-c4dc-4684-bccd-e0674098bdc5" path="/var/lib/kubelet/pods/96a18860-c4dc-4684-bccd-e0674098bdc5/volumes"
Oct 30 22:17:32.058383 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:17:32.058331    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-105770a5-v79d2"]
Oct 30 22:17:32.102290 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:17:32.102260    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-105770a5-v79d2"]
Oct 30 22:17:33.708336 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:17:33.708302    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="724f88e6-a650-46a6-a26b-a812b30d6112" path="/var/lib/kubelet/pods/724f88e6-a650-46a6-a26b-a812b30d6112/volumes"
Oct 30 22:17:47.166660 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:17:47.166578    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:17:55.053238 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:17:55.053204    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f84a2ad-bc4cc"]
Oct 30 22:17:55.102186 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:17:55.102158    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f84a2ad-bc4cc"]
Oct 30 22:17:55.710159 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:17:55.710118    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c30ec77-1b75-482e-a713-0aaff964756f" path="/var/lib/kubelet/pods/6c30ec77-1b75-482e-a713-0aaff964756f/volumes"
Oct 30 22:18:07.655236 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:07.655203    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"]
Oct 30 22:18:17.168846 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:18:17.168801    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:18:22.675830 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:22.675764    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw" podUID="368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357" containerName="main" containerID="cri-o://641fdd7935343a26f8c0471147532f0a875660cc7321b4446a65ec864eacb05a" gracePeriod=45
Oct 30 22:18:22.938161 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:22.938089    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"
Oct 30 22:18:23.045437 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:23.045325    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-kube-api-access-vvfcx\" (UniqueName: \"kubernetes.io/projected/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-kube-api-access-vvfcx\") pod \"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357\" (UID: \"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357\") "
Oct 30 22:18:23.045709 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:23.045566    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-tmp-dir\") pod \"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357\" (UID: \"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357\") "
Oct 30 22:18:23.045828 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:23.045718    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-tmp-dir" pod "368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357" (UID: "368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:23.045947 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:23.045730    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-model-cache\") pod \"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357\" (UID: \"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357\") "
Oct 30 22:18:23.045947 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:23.045880    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-tls-certs\" (UniqueName: \"kubernetes.io/secret/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-tls-certs\") pod \"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357\" (UID: \"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357\") "
Oct 30 22:18:23.045947 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:23.045934    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-dshm\" (UniqueName: \"kubernetes.io/empty-dir/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-dshm\") pod \"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357\" (UID: \"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357\") "
Oct 30 22:18:23.046106 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:23.045998    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-home\" (UniqueName: \"kubernetes.io/empty-dir/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-home\") pod \"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357\" (UID: \"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357\") "
Oct 30 22:18:23.046286 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:23.046264    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-model-cache" pod "368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357" (UID: "368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:23.046426 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:23.046400    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-home" pod "368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357" (UID: "368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:23.046492 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:23.046467    5087 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-model-cache\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:23.046492 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:23.046487    5087 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-tmp-dir\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:23.048736 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:23.048691    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-dshm" pod "368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357" (UID: "368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:23.048962 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:23.048933    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-kube-api-access-vvfcx" pod "368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357" (UID: "368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357"). InnerVolumeSpecName "kube-api-access-vvfcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:23.049198 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:23.049170    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-tls-certs" pod "368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357" (UID: "368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:18:23.147421 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:23.147391    5087 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-tls-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:23.147421 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:23.147414    5087 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-dshm\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:23.147421 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:23.147423    5087 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-home\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:23.147669 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:23.147432    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvfcx\" (UniqueName: \"kubernetes.io/projected/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357-kube-api-access-vvfcx\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:18:23.542953 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:23.542919    5087 generic.go:356] "Generic (PLEG): container finished" podID="368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357" containerID="641fdd7935343a26f8c0471147532f0a875660cc7321b4446a65ec864eacb05a" exitCode=0
Oct 30 22:18:23.543146 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:23.542971    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"
Oct 30 22:18:23.543146 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:23.543011    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw" event={"ID":"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357","Type":"ContainerDied","Data":"641fdd7935343a26f8c0471147532f0a875660cc7321b4446a65ec864eacb05a"}
Oct 30 22:18:23.543146 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:23.543061    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw" event={"ID":"368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357","Type":"ContainerDied","Data":"cbb380428e6b884909353038dba57019a40d37a3c3c95bf122b0f030e747bc82"}
Oct 30 22:18:23.543146 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:23.543081    5087 scope.go:122] "RemoveContainer" containerID="641fdd7935343a26f8c0471147532f0a875660cc7321b4446a65ec864eacb05a"
Oct 30 22:18:23.553282 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:23.553259    5087 scope.go:122] "RemoveContainer" containerID="641fdd7935343a26f8c0471147532f0a875660cc7321b4446a65ec864eacb05a"
Oct 30 22:18:23.553582 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:18:23.553558    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"641fdd7935343a26f8c0471147532f0a875660cc7321b4446a65ec864eacb05a\": container with ID starting with 641fdd7935343a26f8c0471147532f0a875660cc7321b4446a65ec864eacb05a not found: ID does not exist" containerID="641fdd7935343a26f8c0471147532f0a875660cc7321b4446a65ec864eacb05a"
Oct 30 22:18:23.553642 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:23.553592    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"641fdd7935343a26f8c0471147532f0a875660cc7321b4446a65ec864eacb05a"} err="failed to get container status \"641fdd7935343a26f8c0471147532f0a875660cc7321b4446a65ec864eacb05a\": rpc error: code = NotFound desc = could not find container \"641fdd7935343a26f8c0471147532f0a875660cc7321b4446a65ec864eacb05a\": container with ID starting with 641fdd7935343a26f8c0471147532f0a875660cc7321b4446a65ec864eacb05a not found: ID does not exist"
Oct 30 22:18:23.709540 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:23.709513    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"]
Oct 30 22:18:23.711246 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:23.711224    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw"]
Oct 30 22:18:23.874381 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:23.874281    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-9da607/test-model-9da607-kserve-5cdb7cd6d9-l8bsw" podUID="368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357" containerName="main" probeResult="failure" output="Get \"https://10.132.0.207:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:18:25.711617 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:25.711584    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357" path="/var/lib/kubelet/pods/368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357/volumes"
Oct 30 22:18:26.175473 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:26.175439    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 22:18:26.186340 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:26.186308    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 22:18:26.190250 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:26.190225    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 22:18:26.200024 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:26.200003    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 22:18:29.831943 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:29.831915    5087 scope.go:122] "RemoveContainer" containerID="9999eb380c55205e07a5f8fdb9e99ce521bc765973bc430325c99cab4877a9ca"
Oct 30 22:18:29.841431 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:29.841408    5087 scope.go:122] "RemoveContainer" containerID="53786d0863d7e55572ae7dc6870afe201984938bf5b187654c121d704ad97a26"
Oct 30 22:18:29.851646 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:29.851626    5087 scope.go:122] "RemoveContainer" containerID="8e17283b9c1288f08f02f1408f8f968f6503aa741e018c1e9820083646d8e5d2"
Oct 30 22:18:29.861701 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:18:29.861669    5087 scope.go:122] "RemoveContainer" containerID="afabac5215053f697d1ef8032d2ce9c99e1a08643d2b591c0d78ca58adf1bac4"
Oct 30 22:18:47.167779 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:18:47.167739    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:19:09.274540 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:09.274457    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9"]
Oct 30 22:19:09.275056 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:09.274664    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9" podUID="8a031d03-d200-4477-9a61-8385a063abf1" containerName="maas-api" containerID="cri-o://4c8e628dd24c3c80a37135c058fbb3db142aa8808efa98b4dd89e9a2ca981f5b" gracePeriod=30
Oct 30 22:19:09.548161 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:09.548134    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9"
Oct 30 22:19:09.622093 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:09.622064    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a031d03-d200-4477-9a61-8385a063abf1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a031d03-d200-4477-9a61-8385a063abf1-metrics-certs\") pod \"8a031d03-d200-4477-9a61-8385a063abf1\" (UID: \"8a031d03-d200-4477-9a61-8385a063abf1\") "
Oct 30 22:19:09.622253 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:09.622104    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a031d03-d200-4477-9a61-8385a063abf1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a031d03-d200-4477-9a61-8385a063abf1-maas-api-tls\") pod \"8a031d03-d200-4477-9a61-8385a063abf1\" (UID: \"8a031d03-d200-4477-9a61-8385a063abf1\") "
Oct 30 22:19:09.622253 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:09.622162    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a031d03-d200-4477-9a61-8385a063abf1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a031d03-d200-4477-9a61-8385a063abf1-openshift-service-ca\") pod \"8a031d03-d200-4477-9a61-8385a063abf1\" (UID: \"8a031d03-d200-4477-9a61-8385a063abf1\") "
Oct 30 22:19:09.622253 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:09.622241    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a031d03-d200-4477-9a61-8385a063abf1-kube-api-access-lfxzm\" (UniqueName: \"kubernetes.io/projected/8a031d03-d200-4477-9a61-8385a063abf1-kube-api-access-lfxzm\") pod \"8a031d03-d200-4477-9a61-8385a063abf1\" (UID: \"8a031d03-d200-4477-9a61-8385a063abf1\") "
Oct 30 22:19:09.622960 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:09.622651    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a031d03-d200-4477-9a61-8385a063abf1-openshift-service-ca" pod "8a031d03-d200-4477-9a61-8385a063abf1" (UID: "8a031d03-d200-4477-9a61-8385a063abf1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:09.624668 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:09.624624    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a031d03-d200-4477-9a61-8385a063abf1-maas-api-tls" pod "8a031d03-d200-4477-9a61-8385a063abf1" (UID: "8a031d03-d200-4477-9a61-8385a063abf1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:09.625009 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:09.624984    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a031d03-d200-4477-9a61-8385a063abf1-metrics-certs" pod "8a031d03-d200-4477-9a61-8385a063abf1" (UID: "8a031d03-d200-4477-9a61-8385a063abf1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:09.625131 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:09.625095    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a031d03-d200-4477-9a61-8385a063abf1-kube-api-access-lfxzm" pod "8a031d03-d200-4477-9a61-8385a063abf1" (UID: "8a031d03-d200-4477-9a61-8385a063abf1"). InnerVolumeSpecName "kube-api-access-lfxzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:09.723837 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:09.723807    5087 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a031d03-d200-4477-9a61-8385a063abf1-metrics-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:09.723837 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:09.723837    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8a031d03-d200-4477-9a61-8385a063abf1-maas-api-tls\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:09.724066 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:09.723852    5087 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a031d03-d200-4477-9a61-8385a063abf1-openshift-service-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:09.724066 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:09.723866    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfxzm\" (UniqueName: \"kubernetes.io/projected/8a031d03-d200-4477-9a61-8385a063abf1-kube-api-access-lfxzm\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:09.807727 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:09.807637    5087 generic.go:356] "Generic (PLEG): container finished" podID="8a031d03-d200-4477-9a61-8385a063abf1" containerID="4c8e628dd24c3c80a37135c058fbb3db142aa8808efa98b4dd89e9a2ca981f5b" exitCode=0
Oct 30 22:19:09.807727 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:09.807695    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9"
Oct 30 22:19:09.807727 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:09.807701    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9" event={"ID":"8a031d03-d200-4477-9a61-8385a063abf1","Type":"ContainerDied","Data":"4c8e628dd24c3c80a37135c058fbb3db142aa8808efa98b4dd89e9a2ca981f5b"}
Oct 30 22:19:09.808001 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:09.807776    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9" event={"ID":"8a031d03-d200-4477-9a61-8385a063abf1","Type":"ContainerDied","Data":"94ea4c79c4bbd46f9c1e586b40492d4bfa2e24675eb3be87cdfaba4544709929"}
Oct 30 22:19:09.808001 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:09.807807    5087 scope.go:122] "RemoveContainer" containerID="4c8e628dd24c3c80a37135c058fbb3db142aa8808efa98b4dd89e9a2ca981f5b"
Oct 30 22:19:09.818168 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:09.818151    5087 scope.go:122] "RemoveContainer" containerID="4c8e628dd24c3c80a37135c058fbb3db142aa8808efa98b4dd89e9a2ca981f5b"
Oct 30 22:19:09.818497 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:19:09.818477    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c8e628dd24c3c80a37135c058fbb3db142aa8808efa98b4dd89e9a2ca981f5b\": container with ID starting with 4c8e628dd24c3c80a37135c058fbb3db142aa8808efa98b4dd89e9a2ca981f5b not found: ID does not exist" containerID="4c8e628dd24c3c80a37135c058fbb3db142aa8808efa98b4dd89e9a2ca981f5b"
Oct 30 22:19:09.818561 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:09.818506    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c8e628dd24c3c80a37135c058fbb3db142aa8808efa98b4dd89e9a2ca981f5b"} err="failed to get container status \"4c8e628dd24c3c80a37135c058fbb3db142aa8808efa98b4dd89e9a2ca981f5b\": rpc error: code = NotFound desc = could not find container \"4c8e628dd24c3c80a37135c058fbb3db142aa8808efa98b4dd89e9a2ca981f5b\": container with ID starting with 4c8e628dd24c3c80a37135c058fbb3db142aa8808efa98b4dd89e9a2ca981f5b not found: ID does not exist"
Oct 30 22:19:09.971098 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:09.971066    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9"]
Oct 30 22:19:09.975894 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:09.975873    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-9da607-69775fc85f-dqpb9"]
Oct 30 22:19:11.708444 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:11.708410    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a031d03-d200-4477-9a61-8385a063abf1" path="/var/lib/kubelet/pods/8a031d03-d200-4477-9a61-8385a063abf1/volumes"
Oct 30 22:19:17.081403 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:17.081371    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dc4c98b-b52m5"]
Oct 30 22:19:17.130984 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:17.130956    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4ea03cd-zzd66"]
Oct 30 22:19:17.132838 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:17.132820    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1dc4c98b-b52m5"]
Oct 30 22:19:17.134614 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:17.134593    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4ea03cd-zzd66"]
Oct 30 22:19:17.167128 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:19:17.167096    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:19:17.709269 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:17.709236    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f9b70cb-bfbf-40cd-beab-1860e8d63d7a" path="/var/lib/kubelet/pods/4f9b70cb-bfbf-40cd-beab-1860e8d63d7a/volumes"
Oct 30 22:19:17.709749 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:17.709732    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acf0442b-7e18-44ad-b993-65ca686d6013" path="/var/lib/kubelet/pods/acf0442b-7e18-44ad-b993-65ca686d6013/volumes"
Oct 30 22:19:19.734991 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:19.734960    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"]
Oct 30 22:19:19.735426 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:19.735168    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42" podUID="a6676e8e-cec6-431d-8a80-ae56c9843093" containerName="istio-proxy" containerID="cri-o://988a607d076aed36fc14b181972df1c4d2e8d0af44ccc5fada3b8360c91d0b70" gracePeriod=30
Oct 30 22:19:20.682679 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:20.682644    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-954628-9xndd"]
Oct 30 22:19:20.683124 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:20.683110    5087 cpu_manager.go:414] "containerMap: removing container" podUID="ace33c00-1c8a-4626-a72a-4d6e74a86a16" containerName="revoke-keys"
Oct 30 22:19:20.683169 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:20.683126    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace33c00-1c8a-4626-a72a-4d6e74a86a16" containerName="revoke-keys"
Oct 30 22:19:20.683169 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:20.683143    5087 cpu_manager.go:414] "containerMap: removing container" podUID="8a031d03-d200-4477-9a61-8385a063abf1" containerName="maas-api"
Oct 30 22:19:20.683169 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:20.683151    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a031d03-d200-4477-9a61-8385a063abf1" containerName="maas-api"
Oct 30 22:19:20.683271 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:20.683177    5087 cpu_manager.go:414] "containerMap: removing container" podUID="368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357" containerName="main"
Oct 30 22:19:20.683271 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:20.683182    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357" containerName="main"
Oct 30 22:19:20.683271 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:20.683255    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="368c14ec-f6ad-4c3e-bbaa-ca88e9cc1357" containerName="main"
Oct 30 22:19:20.683271 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:20.683267    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace33c00-1c8a-4626-a72a-4d6e74a86a16" containerName="revoke-keys"
Oct 30 22:19:20.683435 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:20.683273    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a031d03-d200-4477-9a61-8385a063abf1" containerName="maas-api"
Oct 30 22:19:20.687921 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:20.687905    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-954628-9xndd"
Oct 30 22:19:20.743485 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:20.743421    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-954628-9xndd"]
Oct 30 22:19:20.789731 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:20.789706    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:19:20.789892 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:20.789822    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:19:20.835240 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:20.835213    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bd22c6d-3dab-42ab-936d-4c6048c51a84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-954628-9xndd\" (UID: \"0bd22c6d-3dab-42ab-936d-4c6048c51a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-954628-9xndd"
Oct 30 22:19:20.835433 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:20.835283    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2swt\" (UniqueName: \"kubernetes.io/projected/0bd22c6d-3dab-42ab-936d-4c6048c51a84-kube-api-access-v2swt\") pod \"maas-api-revoke-keys-e2e-inf-b-954628-9xndd\" (UID: \"0bd22c6d-3dab-42ab-936d-4c6048c51a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-954628-9xndd"
Oct 30 22:19:20.835433 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:20.835394    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bd22c6d-3dab-42ab-936d-4c6048c51a84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-954628-9xndd\" (UID: \"0bd22c6d-3dab-42ab-936d-4c6048c51a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-954628-9xndd"
Oct 30 22:19:20.936548 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:20.936454    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bd22c6d-3dab-42ab-936d-4c6048c51a84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-954628-9xndd\" (UID: \"0bd22c6d-3dab-42ab-936d-4c6048c51a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-954628-9xndd"
Oct 30 22:19:20.936548 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:20.936540    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2swt\" (UniqueName: \"kubernetes.io/projected/0bd22c6d-3dab-42ab-936d-4c6048c51a84-kube-api-access-v2swt\") pod \"maas-api-revoke-keys-e2e-inf-b-954628-9xndd\" (UID: \"0bd22c6d-3dab-42ab-936d-4c6048c51a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-954628-9xndd"
Oct 30 22:19:20.936797 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:20.936565    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bd22c6d-3dab-42ab-936d-4c6048c51a84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-954628-9xndd\" (UID: \"0bd22c6d-3dab-42ab-936d-4c6048c51a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-954628-9xndd"
Oct 30 22:19:20.937105 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:20.937083    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bd22c6d-3dab-42ab-936d-4c6048c51a84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-954628-9xndd\" (UID: \"0bd22c6d-3dab-42ab-936d-4c6048c51a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-954628-9xndd"
Oct 30 22:19:20.939118 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:20.939094    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bd22c6d-3dab-42ab-936d-4c6048c51a84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-954628-9xndd\" (UID: \"0bd22c6d-3dab-42ab-936d-4c6048c51a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-954628-9xndd"
Oct 30 22:19:20.993264 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:20.993238    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2swt\" (UniqueName: \"kubernetes.io/projected/0bd22c6d-3dab-42ab-936d-4c6048c51a84-kube-api-access-v2swt\") pod \"maas-api-revoke-keys-e2e-inf-b-954628-9xndd\" (UID: \"0bd22c6d-3dab-42ab-936d-4c6048c51a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-954628-9xndd"
Oct 30 22:19:21.001162 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:21.001116    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-954628-9xndd"
Oct 30 22:19:21.374788 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:21.374763    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-954628-9xndd"]
Oct 30 22:19:21.376932 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:19:21.376903    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd22c6d_3dab_42ab_936d_4c6048c51a84.slice/crio-59c4f78cf2ab805d49186657667d0af4c8b79aae7862fd5df9a3a8574a01162d WatchSource:0}: Error finding container 59c4f78cf2ab805d49186657667d0af4c8b79aae7862fd5df9a3a8574a01162d: Status 404 returned error can't find the container with id 59c4f78cf2ab805d49186657667d0af4c8b79aae7862fd5df9a3a8574a01162d
Oct 30 22:19:21.879838 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:21.879801    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-954628-9xndd" event={"ID":"0bd22c6d-3dab-42ab-936d-4c6048c51a84","Type":"ContainerStarted","Data":"b3191aec0d7d315131b7726bd9f59a229ed694b1b6302e308271a638807f34e2"}
Oct 30 22:19:21.880243 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:21.879858    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-954628-9xndd" event={"ID":"0bd22c6d-3dab-42ab-936d-4c6048c51a84","Type":"ContainerStarted","Data":"59c4f78cf2ab805d49186657667d0af4c8b79aae7862fd5df9a3a8574a01162d"}
Oct 30 22:19:21.994171 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:21.994102    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-954628-9xndd" podStartSLOduration=1.994084983 podStartE2EDuration="1.994084983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:21.993166905 +0000 UTC m=+2756.467670880" watchObservedRunningTime="2026-10-30 22:19:21.994084983 +0000 UTC m=+2756.468588965"
Oct 30 22:19:22.889133 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:22.889098    5087 generic.go:356] "Generic (PLEG): container finished" podID="0bd22c6d-3dab-42ab-936d-4c6048c51a84" containerID="b3191aec0d7d315131b7726bd9f59a229ed694b1b6302e308271a638807f34e2" exitCode=0
Oct 30 22:19:22.889560 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:22.889136    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-954628-9xndd" event={"ID":"0bd22c6d-3dab-42ab-936d-4c6048c51a84","Type":"ContainerDied","Data":"b3191aec0d7d315131b7726bd9f59a229ed694b1b6302e308271a638807f34e2"}
Oct 30 22:19:23.237223 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:23.237188    5087 patch_prober.go:29] interesting pod/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:19:23.237399 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:23.237254    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42" podUID="a6676e8e-cec6-431d-8a80-ae56c9843093" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:19:24.038989 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:24.038966    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-954628-9xndd"
Oct 30 22:19:24.172415 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:24.172325    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bd22c6d-3dab-42ab-936d-4c6048c51a84-kube-api-access-v2swt\" (UniqueName: \"kubernetes.io/projected/0bd22c6d-3dab-42ab-936d-4c6048c51a84-kube-api-access-v2swt\") pod \"0bd22c6d-3dab-42ab-936d-4c6048c51a84\" (UID: \"0bd22c6d-3dab-42ab-936d-4c6048c51a84\") "
Oct 30 22:19:24.172581 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:24.172520    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bd22c6d-3dab-42ab-936d-4c6048c51a84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bd22c6d-3dab-42ab-936d-4c6048c51a84-maas-api-xks-ca\") pod \"0bd22c6d-3dab-42ab-936d-4c6048c51a84\" (UID: \"0bd22c6d-3dab-42ab-936d-4c6048c51a84\") "
Oct 30 22:19:24.172657 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:24.172597    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bd22c6d-3dab-42ab-936d-4c6048c51a84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bd22c6d-3dab-42ab-936d-4c6048c51a84-maas-api-openshift-ca\") pod \"0bd22c6d-3dab-42ab-936d-4c6048c51a84\" (UID: \"0bd22c6d-3dab-42ab-936d-4c6048c51a84\") "
Oct 30 22:19:24.172982 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:24.172946    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bd22c6d-3dab-42ab-936d-4c6048c51a84-maas-api-openshift-ca" pod "0bd22c6d-3dab-42ab-936d-4c6048c51a84" (UID: "0bd22c6d-3dab-42ab-936d-4c6048c51a84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:24.174937 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:24.174906    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bd22c6d-3dab-42ab-936d-4c6048c51a84-maas-api-xks-ca" pod "0bd22c6d-3dab-42ab-936d-4c6048c51a84" (UID: "0bd22c6d-3dab-42ab-936d-4c6048c51a84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:24.175251 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:24.175226    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd22c6d-3dab-42ab-936d-4c6048c51a84-kube-api-access-v2swt" pod "0bd22c6d-3dab-42ab-936d-4c6048c51a84" (UID: "0bd22c6d-3dab-42ab-936d-4c6048c51a84"). InnerVolumeSpecName "kube-api-access-v2swt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:24.274239 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:24.274207    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bd22c6d-3dab-42ab-936d-4c6048c51a84-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:24.274239 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:24.274232    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bd22c6d-3dab-42ab-936d-4c6048c51a84-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:24.274239 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:24.274243    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2swt\" (UniqueName: \"kubernetes.io/projected/0bd22c6d-3dab-42ab-936d-4c6048c51a84-kube-api-access-v2swt\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:24.902551 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:24.902513    5087 generic.go:356] "Generic (PLEG): container finished" podID="a6676e8e-cec6-431d-8a80-ae56c9843093" containerID="988a607d076aed36fc14b181972df1c4d2e8d0af44ccc5fada3b8360c91d0b70" exitCode=0
Oct 30 22:19:24.902741 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:24.902593    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42" event={"ID":"a6676e8e-cec6-431d-8a80-ae56c9843093","Type":"ContainerDied","Data":"988a607d076aed36fc14b181972df1c4d2e8d0af44ccc5fada3b8360c91d0b70"}
Oct 30 22:19:24.902741 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:24.902669    5087 scope.go:122] "RemoveContainer" containerID="f767459022cd88fdbd9239afc6d4af710065bab5b3b1a5871a19dfb45e388d3c"
Oct 30 22:19:24.904411 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:24.904383    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-954628-9xndd" event={"ID":"0bd22c6d-3dab-42ab-936d-4c6048c51a84","Type":"ContainerDied","Data":"59c4f78cf2ab805d49186657667d0af4c8b79aae7862fd5df9a3a8574a01162d"}
Oct 30 22:19:24.904522 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:24.904434    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59c4f78cf2ab805d49186657667d0af4c8b79aae7862fd5df9a3a8574a01162d"
Oct 30 22:19:24.904522 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:24.904453    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-954628-9xndd"
Oct 30 22:19:24.986095 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:24.986069    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:19:25.082653 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:25.082620    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6676e8e-cec6-431d-8a80-ae56c9843093-kube-api-access-d4r9g\" (UniqueName: \"kubernetes.io/projected/a6676e8e-cec6-431d-8a80-ae56c9843093-kube-api-access-d4r9g\") pod \"a6676e8e-cec6-431d-8a80-ae56c9843093\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") "
Oct 30 22:19:25.083139 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:25.082666    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-istio-envoy\") pod \"a6676e8e-cec6-431d-8a80-ae56c9843093\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") "
Oct 30 22:19:25.083139 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:25.082701    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6676e8e-cec6-431d-8a80-ae56c9843093-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6676e8e-cec6-431d-8a80-ae56c9843093-istio-podinfo\") pod \"a6676e8e-cec6-431d-8a80-ae56c9843093\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") "
Oct 30 22:19:25.083139 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:25.082771    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6676e8e-cec6-431d-8a80-ae56c9843093-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6676e8e-cec6-431d-8a80-ae56c9843093-istiod-ca-cert\") pod \"a6676e8e-cec6-431d-8a80-ae56c9843093\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") "
Oct 30 22:19:25.083139 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:25.082804    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-workload-certs\") pod \"a6676e8e-cec6-431d-8a80-ae56c9843093\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") "
Oct 30 22:19:25.083139 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:25.082842    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-istio-data\") pod \"a6676e8e-cec6-431d-8a80-ae56c9843093\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") "
Oct 30 22:19:25.083139 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:25.082902    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-credential-socket\") pod \"a6676e8e-cec6-431d-8a80-ae56c9843093\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") "
Oct 30 22:19:25.083139 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:25.082973    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6676e8e-cec6-431d-8a80-ae56c9843093-istio-token\" (UniqueName: \"kubernetes.io/projected/a6676e8e-cec6-431d-8a80-ae56c9843093-istio-token\") pod \"a6676e8e-cec6-431d-8a80-ae56c9843093\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") "
Oct 30 22:19:25.083139 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:25.083010    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-workload-socket\") pod \"a6676e8e-cec6-431d-8a80-ae56c9843093\" (UID: \"a6676e8e-cec6-431d-8a80-ae56c9843093\") "
Oct 30 22:19:25.083139 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:25.083119    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-istio-data" pod "a6676e8e-cec6-431d-8a80-ae56c9843093" (UID: "a6676e8e-cec6-431d-8a80-ae56c9843093"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:25.083773 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:25.083297    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-workload-certs" pod "a6676e8e-cec6-431d-8a80-ae56c9843093" (UID: "a6676e8e-cec6-431d-8a80-ae56c9843093"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:25.083773 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:25.083316    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:25.083773 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:25.083339    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6676e8e-cec6-431d-8a80-ae56c9843093-istiod-ca-cert" pod "a6676e8e-cec6-431d-8a80-ae56c9843093" (UID: "a6676e8e-cec6-431d-8a80-ae56c9843093"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:25.083773 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:25.083535    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-credential-socket" pod "a6676e8e-cec6-431d-8a80-ae56c9843093" (UID: "a6676e8e-cec6-431d-8a80-ae56c9843093"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:25.083773 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:25.083564    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-workload-socket" pod "a6676e8e-cec6-431d-8a80-ae56c9843093" (UID: "a6676e8e-cec6-431d-8a80-ae56c9843093"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:25.085505 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:25.085481    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-istio-envoy" pod "a6676e8e-cec6-431d-8a80-ae56c9843093" (UID: "a6676e8e-cec6-431d-8a80-ae56c9843093"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:25.085813 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:25.085785    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6676e8e-cec6-431d-8a80-ae56c9843093-istio-podinfo" pod "a6676e8e-cec6-431d-8a80-ae56c9843093" (UID: "a6676e8e-cec6-431d-8a80-ae56c9843093"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:19:25.085930 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:25.085841    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6676e8e-cec6-431d-8a80-ae56c9843093-kube-api-access-d4r9g" pod "a6676e8e-cec6-431d-8a80-ae56c9843093" (UID: "a6676e8e-cec6-431d-8a80-ae56c9843093"). InnerVolumeSpecName "kube-api-access-d4r9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:25.086002 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:25.085943    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6676e8e-cec6-431d-8a80-ae56c9843093-istio-token" pod "a6676e8e-cec6-431d-8a80-ae56c9843093" (UID: "a6676e8e-cec6-431d-8a80-ae56c9843093"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:25.184083 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:25.184047    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4r9g\" (UniqueName: \"kubernetes.io/projected/a6676e8e-cec6-431d-8a80-ae56c9843093-kube-api-access-d4r9g\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:25.184083 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:25.184075    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:25.184372 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:25.184092    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6676e8e-cec6-431d-8a80-ae56c9843093-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:25.184372 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:25.184105    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6676e8e-cec6-431d-8a80-ae56c9843093-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:25.184372 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:25.184114    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:25.184372 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:25.184123    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:25.184372 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:25.184131    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6676e8e-cec6-431d-8a80-ae56c9843093-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:25.184372 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:25.184141    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6676e8e-cec6-431d-8a80-ae56c9843093-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:25.911381 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:25.911325    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"
Oct 30 22:19:25.911579 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:25.911396    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42" event={"ID":"a6676e8e-cec6-431d-8a80-ae56c9843093","Type":"ContainerDied","Data":"a93c1c7227b0ec2a2ad31601972238d96ecba19bf60e84fc4d518e7d4be4ad97"}
Oct 30 22:19:25.911579 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:25.911443    5087 scope.go:122] "RemoveContainer" containerID="988a607d076aed36fc14b181972df1c4d2e8d0af44ccc5fada3b8360c91d0b70"
Oct 30 22:19:26.140426 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:26.140393    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"]
Oct 30 22:19:26.166266 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:26.166232    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9da607-openshift-default-5d68dbc6c7-jld42"]
Oct 30 22:19:27.709986 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:27.709954    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6676e8e-cec6-431d-8a80-ae56c9843093" path="/var/lib/kubelet/pods/a6676e8e-cec6-431d-8a80-ae56c9843093/volumes"
Oct 30 22:19:29.906241 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:29.906210    5087 scope.go:122] "RemoveContainer" containerID="046e43d9218239342990af0b656c22d9bee82c5c38d7a01a26c09b88386ac131"
Oct 30 22:19:29.915186 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:29.915167    5087 scope.go:122] "RemoveContainer" containerID="96020f9af74e9c23825fb0e96b3c6a87d867bcf5af41c4d3700720239a87c48d"
Oct 30 22:19:29.925819 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:29.925791    5087 scope.go:122] "RemoveContainer" containerID="ef6aa7205da8bb246fcfe0dd9c02fde7813a71bcdea1351c9b227a2cc9929eda"
Oct 30 22:19:35.036367 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:35.036311    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb"]
Oct 30 22:19:35.036852 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:35.036597    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb" podUID="fe364587-c942-40b4-ae87-10c7cc1176b4" containerName="maas-api" containerID="cri-o://3c8dd812c0543cd619e975e8589844bf0ccfb918466ef5f47e7a942b6cdbaa4e" gracePeriod=30
Oct 30 22:19:35.314937 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:35.314912    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb"
Oct 30 22:19:35.385002 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:35.384971    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe364587-c942-40b4-ae87-10c7cc1176b4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe364587-c942-40b4-ae87-10c7cc1176b4-openshift-service-ca\") pod \"fe364587-c942-40b4-ae87-10c7cc1176b4\" (UID: \"fe364587-c942-40b4-ae87-10c7cc1176b4\") "
Oct 30 22:19:35.385002 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:35.385005    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe364587-c942-40b4-ae87-10c7cc1176b4-kube-api-access-h9nzm\" (UniqueName: \"kubernetes.io/projected/fe364587-c942-40b4-ae87-10c7cc1176b4-kube-api-access-h9nzm\") pod \"fe364587-c942-40b4-ae87-10c7cc1176b4\" (UID: \"fe364587-c942-40b4-ae87-10c7cc1176b4\") "
Oct 30 22:19:35.385278 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:35.385043    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe364587-c942-40b4-ae87-10c7cc1176b4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe364587-c942-40b4-ae87-10c7cc1176b4-maas-api-tls\") pod \"fe364587-c942-40b4-ae87-10c7cc1176b4\" (UID: \"fe364587-c942-40b4-ae87-10c7cc1176b4\") "
Oct 30 22:19:35.385278 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:35.385189    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe364587-c942-40b4-ae87-10c7cc1176b4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe364587-c942-40b4-ae87-10c7cc1176b4-metrics-certs\") pod \"fe364587-c942-40b4-ae87-10c7cc1176b4\" (UID: \"fe364587-c942-40b4-ae87-10c7cc1176b4\") "
Oct 30 22:19:35.385510 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:35.385471    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe364587-c942-40b4-ae87-10c7cc1176b4-openshift-service-ca" pod "fe364587-c942-40b4-ae87-10c7cc1176b4" (UID: "fe364587-c942-40b4-ae87-10c7cc1176b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:35.387425 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:35.387389    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe364587-c942-40b4-ae87-10c7cc1176b4-maas-api-tls" pod "fe364587-c942-40b4-ae87-10c7cc1176b4" (UID: "fe364587-c942-40b4-ae87-10c7cc1176b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:35.387584 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:35.387564    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe364587-c942-40b4-ae87-10c7cc1176b4-metrics-certs" pod "fe364587-c942-40b4-ae87-10c7cc1176b4" (UID: "fe364587-c942-40b4-ae87-10c7cc1176b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:35.387635 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:35.387571    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe364587-c942-40b4-ae87-10c7cc1176b4-kube-api-access-h9nzm" pod "fe364587-c942-40b4-ae87-10c7cc1176b4" (UID: "fe364587-c942-40b4-ae87-10c7cc1176b4"). InnerVolumeSpecName "kube-api-access-h9nzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:35.486917 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:35.486869    5087 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe364587-c942-40b4-ae87-10c7cc1176b4-openshift-service-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:35.486917 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:35.486901    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9nzm\" (UniqueName: \"kubernetes.io/projected/fe364587-c942-40b4-ae87-10c7cc1176b4-kube-api-access-h9nzm\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:35.486917 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:35.486913    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fe364587-c942-40b4-ae87-10c7cc1176b4-maas-api-tls\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:35.486917 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:35.486922    5087 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe364587-c942-40b4-ae87-10c7cc1176b4-metrics-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:35.976184 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:35.976054    5087 generic.go:356] "Generic (PLEG): container finished" podID="fe364587-c942-40b4-ae87-10c7cc1176b4" containerID="3c8dd812c0543cd619e975e8589844bf0ccfb918466ef5f47e7a942b6cdbaa4e" exitCode=0
Oct 30 22:19:35.976184 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:35.976142    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb"
Oct 30 22:19:35.976184 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:35.976145    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb" event={"ID":"fe364587-c942-40b4-ae87-10c7cc1176b4","Type":"ContainerDied","Data":"3c8dd812c0543cd619e975e8589844bf0ccfb918466ef5f47e7a942b6cdbaa4e"}
Oct 30 22:19:35.976562 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:35.976211    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb" event={"ID":"fe364587-c942-40b4-ae87-10c7cc1176b4","Type":"ContainerDied","Data":"1f48943c3119aec9838e6ccdea48e74a0a1b5adf6f370d1741a8d0c20c71ab2b"}
Oct 30 22:19:35.976562 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:35.976238    5087 scope.go:122] "RemoveContainer" containerID="3c8dd812c0543cd619e975e8589844bf0ccfb918466ef5f47e7a942b6cdbaa4e"
Oct 30 22:19:35.990105 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:35.990072    5087 scope.go:122] "RemoveContainer" containerID="3c8dd812c0543cd619e975e8589844bf0ccfb918466ef5f47e7a942b6cdbaa4e"
Oct 30 22:19:35.991103 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:19:35.991071    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c8dd812c0543cd619e975e8589844bf0ccfb918466ef5f47e7a942b6cdbaa4e\": container with ID starting with 3c8dd812c0543cd619e975e8589844bf0ccfb918466ef5f47e7a942b6cdbaa4e not found: ID does not exist" containerID="3c8dd812c0543cd619e975e8589844bf0ccfb918466ef5f47e7a942b6cdbaa4e"
Oct 30 22:19:35.991341 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:35.991294    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c8dd812c0543cd619e975e8589844bf0ccfb918466ef5f47e7a942b6cdbaa4e"} err="failed to get container status \"3c8dd812c0543cd619e975e8589844bf0ccfb918466ef5f47e7a942b6cdbaa4e\": rpc error: code = NotFound desc = could not find container \"3c8dd812c0543cd619e975e8589844bf0ccfb918466ef5f47e7a942b6cdbaa4e\": container with ID starting with 3c8dd812c0543cd619e975e8589844bf0ccfb918466ef5f47e7a942b6cdbaa4e not found: ID does not exist"
Oct 30 22:19:36.138992 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:36.138958    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb"]
Oct 30 22:19:36.188484 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:36.188452    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-954628-78bdbd498f-6b8hb"]
Oct 30 22:19:37.194220 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:37.194188    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-954628-9xndd"]
Oct 30 22:19:37.243492 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:37.243461    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-954628-9xndd"]
Oct 30 22:19:37.708625 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:37.708583    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bd22c6d-3dab-42ab-936d-4c6048c51a84" path="/var/lib/kubelet/pods/0bd22c6d-3dab-42ab-936d-4c6048c51a84/volumes"
Oct 30 22:19:37.709119 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:37.709097    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe364587-c942-40b4-ae87-10c7cc1176b4" path="/var/lib/kubelet/pods/fe364587-c942-40b4-ae87-10c7cc1176b4/volumes"
Oct 30 22:19:47.167172 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:19:47.167138    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:19:55.198749 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:55.198719    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-326d76-fmf6g"]
Oct 30 22:19:55.199154 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:55.199134    5087 cpu_manager.go:414] "containerMap: removing container" podUID="0bd22c6d-3dab-42ab-936d-4c6048c51a84" containerName="revoke-keys"
Oct 30 22:19:55.199154 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:55.199144    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bd22c6d-3dab-42ab-936d-4c6048c51a84" containerName="revoke-keys"
Oct 30 22:19:55.199229 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:55.199157    5087 cpu_manager.go:414] "containerMap: removing container" podUID="a6676e8e-cec6-431d-8a80-ae56c9843093" containerName="istio-proxy"
Oct 30 22:19:55.199229 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:55.199162    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6676e8e-cec6-431d-8a80-ae56c9843093" containerName="istio-proxy"
Oct 30 22:19:55.199229 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:55.199188    5087 cpu_manager.go:414] "containerMap: removing container" podUID="fe364587-c942-40b4-ae87-10c7cc1176b4" containerName="maas-api"
Oct 30 22:19:55.199229 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:55.199193    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe364587-c942-40b4-ae87-10c7cc1176b4" containerName="maas-api"
Oct 30 22:19:55.199385 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:55.199254    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe364587-c942-40b4-ae87-10c7cc1176b4" containerName="maas-api"
Oct 30 22:19:55.199385 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:55.199266    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bd22c6d-3dab-42ab-936d-4c6048c51a84" containerName="revoke-keys"
Oct 30 22:19:55.199385 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:55.199272    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6676e8e-cec6-431d-8a80-ae56c9843093" containerName="istio-proxy"
Oct 30 22:19:55.199385 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:55.199280    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6676e8e-cec6-431d-8a80-ae56c9843093" containerName="istio-proxy"
Oct 30 22:19:55.202450 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:55.202434    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-326d76-fmf6g"
Oct 30 22:19:55.258317 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:55.258286    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-326d76-fmf6g"]
Oct 30 22:19:55.286964 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:55.286931    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003442c5-8e28-4bab-abeb-ea7ad1048434-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-326d76-fmf6g\" (UID: \"003442c5-8e28-4bab-abeb-ea7ad1048434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-326d76-fmf6g"
Oct 30 22:19:55.287122 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:55.287051    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m25lt\" (UniqueName: \"kubernetes.io/projected/003442c5-8e28-4bab-abeb-ea7ad1048434-kube-api-access-m25lt\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-326d76-fmf6g\" (UID: \"003442c5-8e28-4bab-abeb-ea7ad1048434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-326d76-fmf6g"
Oct 30 22:19:55.287122 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:55.287075    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003442c5-8e28-4bab-abeb-ea7ad1048434-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-326d76-fmf6g\" (UID: \"003442c5-8e28-4bab-abeb-ea7ad1048434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-326d76-fmf6g"
Oct 30 22:19:55.305221 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:55.305196    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:19:55.305400 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:55.305307    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:19:55.388556 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:55.388521    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003442c5-8e28-4bab-abeb-ea7ad1048434-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-326d76-fmf6g\" (UID: \"003442c5-8e28-4bab-abeb-ea7ad1048434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-326d76-fmf6g"
Oct 30 22:19:55.388738 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:55.388605    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m25lt\" (UniqueName: \"kubernetes.io/projected/003442c5-8e28-4bab-abeb-ea7ad1048434-kube-api-access-m25lt\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-326d76-fmf6g\" (UID: \"003442c5-8e28-4bab-abeb-ea7ad1048434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-326d76-fmf6g"
Oct 30 22:19:55.388738 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:55.388636    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003442c5-8e28-4bab-abeb-ea7ad1048434-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-326d76-fmf6g\" (UID: \"003442c5-8e28-4bab-abeb-ea7ad1048434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-326d76-fmf6g"
Oct 30 22:19:55.389153 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:55.389126    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003442c5-8e28-4bab-abeb-ea7ad1048434-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-326d76-fmf6g\" (UID: \"003442c5-8e28-4bab-abeb-ea7ad1048434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-326d76-fmf6g"
Oct 30 22:19:55.391129 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:55.391107    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003442c5-8e28-4bab-abeb-ea7ad1048434-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-326d76-fmf6g\" (UID: \"003442c5-8e28-4bab-abeb-ea7ad1048434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-326d76-fmf6g"
Oct 30 22:19:55.445087 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:55.445058    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m25lt\" (UniqueName: \"kubernetes.io/projected/003442c5-8e28-4bab-abeb-ea7ad1048434-kube-api-access-m25lt\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-326d76-fmf6g\" (UID: \"003442c5-8e28-4bab-abeb-ea7ad1048434\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-326d76-fmf6g"
Oct 30 22:19:55.515920 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:55.515897    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-326d76-fmf6g"
Oct 30 22:19:55.885805 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:55.885782    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-326d76-fmf6g"]
Oct 30 22:19:55.887781 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:19:55.887754    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003442c5_8e28_4bab_abeb_ea7ad1048434.slice/crio-2b009247976d59d7172dd6a7024a98709092b7b5c61a9a32736d2e13db494b31 WatchSource:0}: Error finding container 2b009247976d59d7172dd6a7024a98709092b7b5c61a9a32736d2e13db494b31: Status 404 returned error can't find the container with id 2b009247976d59d7172dd6a7024a98709092b7b5c61a9a32736d2e13db494b31
Oct 30 22:19:56.084077 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:56.084045    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-326d76-fmf6g" event={"ID":"003442c5-8e28-4bab-abeb-ea7ad1048434","Type":"ContainerStarted","Data":"2b009247976d59d7172dd6a7024a98709092b7b5c61a9a32736d2e13db494b31"}
Oct 30 22:19:57.091098 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:57.091067    5087 generic.go:356] "Generic (PLEG): container finished" podID="003442c5-8e28-4bab-abeb-ea7ad1048434" containerID="d739cf8766b04c6aedb4967d48be9eba87236a4e94df7b95c2476d50d3d55b0f" exitCode=0
Oct 30 22:19:57.091530 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:57.091110    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-326d76-fmf6g" event={"ID":"003442c5-8e28-4bab-abeb-ea7ad1048434","Type":"ContainerDied","Data":"d739cf8766b04c6aedb4967d48be9eba87236a4e94df7b95c2476d50d3d55b0f"}
Oct 30 22:19:58.234992 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:58.234969    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-326d76-fmf6g"
Oct 30 22:19:58.319122 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:58.319093    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/003442c5-8e28-4bab-abeb-ea7ad1048434-kube-api-access-m25lt\" (UniqueName: \"kubernetes.io/projected/003442c5-8e28-4bab-abeb-ea7ad1048434-kube-api-access-m25lt\") pod \"003442c5-8e28-4bab-abeb-ea7ad1048434\" (UID: \"003442c5-8e28-4bab-abeb-ea7ad1048434\") "
Oct 30 22:19:58.319305 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:58.319146    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/003442c5-8e28-4bab-abeb-ea7ad1048434-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003442c5-8e28-4bab-abeb-ea7ad1048434-maas-api-openshift-ca\") pod \"003442c5-8e28-4bab-abeb-ea7ad1048434\" (UID: \"003442c5-8e28-4bab-abeb-ea7ad1048434\") "
Oct 30 22:19:58.319305 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:58.319200    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/003442c5-8e28-4bab-abeb-ea7ad1048434-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003442c5-8e28-4bab-abeb-ea7ad1048434-maas-api-xks-ca\") pod \"003442c5-8e28-4bab-abeb-ea7ad1048434\" (UID: \"003442c5-8e28-4bab-abeb-ea7ad1048434\") "
Oct 30 22:19:58.319583 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:58.319562    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/003442c5-8e28-4bab-abeb-ea7ad1048434-maas-api-openshift-ca" pod "003442c5-8e28-4bab-abeb-ea7ad1048434" (UID: "003442c5-8e28-4bab-abeb-ea7ad1048434"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:58.321505 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:58.321481    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/003442c5-8e28-4bab-abeb-ea7ad1048434-maas-api-xks-ca" pod "003442c5-8e28-4bab-abeb-ea7ad1048434" (UID: "003442c5-8e28-4bab-abeb-ea7ad1048434"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:58.321597 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:58.321506    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/003442c5-8e28-4bab-abeb-ea7ad1048434-kube-api-access-m25lt" pod "003442c5-8e28-4bab-abeb-ea7ad1048434" (UID: "003442c5-8e28-4bab-abeb-ea7ad1048434"). InnerVolumeSpecName "kube-api-access-m25lt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:58.420993 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:58.420912    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003442c5-8e28-4bab-abeb-ea7ad1048434-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:58.420993 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:58.420941    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003442c5-8e28-4bab-abeb-ea7ad1048434-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:58.420993 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:58.420954    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m25lt\" (UniqueName: \"kubernetes.io/projected/003442c5-8e28-4bab-abeb-ea7ad1048434-kube-api-access-m25lt\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:19:59.103356 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:59.103313    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-326d76-fmf6g"
Oct 30 22:19:59.103356 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:59.103321    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-326d76-fmf6g" event={"ID":"003442c5-8e28-4bab-abeb-ea7ad1048434","Type":"ContainerDied","Data":"2b009247976d59d7172dd6a7024a98709092b7b5c61a9a32736d2e13db494b31"}
Oct 30 22:19:59.103585 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:19:59.103402    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b009247976d59d7172dd6a7024a98709092b7b5c61a9a32736d2e13db494b31"
Oct 30 22:20:17.167972 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:20:17.167933    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:20:17.527783 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:17.527742    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-326d76-fmf6g"]
Oct 30 22:20:17.578358 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:17.578306    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-326d76-fmf6g"]
Oct 30 22:20:17.708046 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:17.708014    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="003442c5-8e28-4bab-abeb-ea7ad1048434" path="/var/lib/kubelet/pods/003442c5-8e28-4bab-abeb-ea7ad1048434/volumes"
Oct 30 22:20:25.364949 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:25.364915    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-20c7c7-j8jcp"]
Oct 30 22:20:25.365484 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:25.365464    5087 cpu_manager.go:414] "containerMap: removing container" podUID="003442c5-8e28-4bab-abeb-ea7ad1048434" containerName="revoke-keys"
Oct 30 22:20:25.365545 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:25.365486    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="003442c5-8e28-4bab-abeb-ea7ad1048434" containerName="revoke-keys"
Oct 30 22:20:25.365545 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:25.365516    5087 cpu_manager.go:414] "containerMap: removing container" podUID="a6676e8e-cec6-431d-8a80-ae56c9843093" containerName="istio-proxy"
Oct 30 22:20:25.365545 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:25.365521    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6676e8e-cec6-431d-8a80-ae56c9843093" containerName="istio-proxy"
Oct 30 22:20:25.365656 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:25.365591    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="003442c5-8e28-4bab-abeb-ea7ad1048434" containerName="revoke-keys"
Oct 30 22:20:25.369035 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:25.369014    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-20c7c7-j8jcp"
Oct 30 22:20:25.422910 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:25.422846    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:20:25.472120 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:25.472088    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:20:25.474389 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:25.474365    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-20c7c7-j8jcp"]
Oct 30 22:20:25.491642 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:25.491611    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-20c7c7-j8jcp\" (UID: \"82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-20c7c7-j8jcp"
Oct 30 22:20:25.491774 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:25.491664    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-20c7c7-j8jcp\" (UID: \"82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-20c7c7-j8jcp"
Oct 30 22:20:25.491836 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:25.491790    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55n84\" (UniqueName: \"kubernetes.io/projected/82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5-kube-api-access-55n84\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-20c7c7-j8jcp\" (UID: \"82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-20c7c7-j8jcp"
Oct 30 22:20:25.593159 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:25.593130    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-20c7c7-j8jcp\" (UID: \"82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-20c7c7-j8jcp"
Oct 30 22:20:25.593397 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:25.593187    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-20c7c7-j8jcp\" (UID: \"82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-20c7c7-j8jcp"
Oct 30 22:20:25.593397 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:25.593256    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55n84\" (UniqueName: \"kubernetes.io/projected/82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5-kube-api-access-55n84\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-20c7c7-j8jcp\" (UID: \"82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-20c7c7-j8jcp"
Oct 30 22:20:25.593805 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:25.593782    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-20c7c7-j8jcp\" (UID: \"82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-20c7c7-j8jcp"
Oct 30 22:20:25.595902 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:25.595871    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-20c7c7-j8jcp\" (UID: \"82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-20c7c7-j8jcp"
Oct 30 22:20:25.650288 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:25.650226    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55n84\" (UniqueName: \"kubernetes.io/projected/82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5-kube-api-access-55n84\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-20c7c7-j8jcp\" (UID: \"82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-20c7c7-j8jcp"
Oct 30 22:20:25.681615 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:25.681587    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-20c7c7-j8jcp"
Oct 30 22:20:26.056115 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:26.056087    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-20c7c7-j8jcp"]
Oct 30 22:20:26.058373 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:20:26.058323    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82bd9090_7caf_4bf0_a6b6_480cc4a3e2f5.slice/crio-11ef8a8b7959ed2538174c3fd9a2ebbfa431b14b9481b7f57aecd244cbddc2fe WatchSource:0}: Error finding container 11ef8a8b7959ed2538174c3fd9a2ebbfa431b14b9481b7f57aecd244cbddc2fe: Status 404 returned error can't find the container with id 11ef8a8b7959ed2538174c3fd9a2ebbfa431b14b9481b7f57aecd244cbddc2fe
Oct 30 22:20:26.256949 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:26.256914    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-20c7c7-j8jcp" event={"ID":"82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5","Type":"ContainerStarted","Data":"11ef8a8b7959ed2538174c3fd9a2ebbfa431b14b9481b7f57aecd244cbddc2fe"}
Oct 30 22:20:27.264888 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:27.264854    5087 generic.go:356] "Generic (PLEG): container finished" podID="82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5" containerID="84133266bc2f128e656785559d6e72ff02c81e9a5e56385876378213565aec9f" exitCode=0
Oct 30 22:20:27.265274 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:27.264946    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-20c7c7-j8jcp" event={"ID":"82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5","Type":"ContainerDied","Data":"84133266bc2f128e656785559d6e72ff02c81e9a5e56385876378213565aec9f"}
Oct 30 22:20:28.424296 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:28.424269    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-20c7c7-j8jcp"
Oct 30 22:20:28.521705 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:28.521678    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5-kube-api-access-55n84\" (UniqueName: \"kubernetes.io/projected/82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5-kube-api-access-55n84\") pod \"82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5\" (UID: \"82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5\") "
Oct 30 22:20:28.521906 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:28.521816    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5-maas-api-openshift-ca\") pod \"82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5\" (UID: \"82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5\") "
Oct 30 22:20:28.521906 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:28.521862    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5-maas-api-xks-ca\") pod \"82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5\" (UID: \"82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5\") "
Oct 30 22:20:28.522202 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:28.522179    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5-maas-api-openshift-ca" pod "82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5" (UID: "82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:20:28.524018 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:28.523966    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5-maas-api-xks-ca" pod "82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5" (UID: "82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:20:28.524091 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:28.524056    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5-kube-api-access-55n84" pod "82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5" (UID: "82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5"). InnerVolumeSpecName "kube-api-access-55n84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:20:28.622894 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:28.622866    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:20:28.622894 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:28.622893    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:20:28.622894 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:28.622903    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55n84\" (UniqueName: \"kubernetes.io/projected/82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5-kube-api-access-55n84\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:20:29.280690 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:29.280662    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-20c7c7-j8jcp"
Oct 30 22:20:29.280858 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:29.280696    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-20c7c7-j8jcp" event={"ID":"82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5","Type":"ContainerDied","Data":"11ef8a8b7959ed2538174c3fd9a2ebbfa431b14b9481b7f57aecd244cbddc2fe"}
Oct 30 22:20:29.280858 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:29.280775    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11ef8a8b7959ed2538174c3fd9a2ebbfa431b14b9481b7f57aecd244cbddc2fe"
Oct 30 22:20:29.964195 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:29.964165    5087 scope.go:122] "RemoveContainer" containerID="778060b48a4e289187ebfa73e8d77c0546f4118d80f8f9c1d20227b53aed4f64"
Oct 30 22:20:29.973568 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:29.973543    5087 scope.go:122] "RemoveContainer" containerID="819e9b07f60279c3bf7772e1a561651b93b0d9e739c3e5ac33d93315895af2e0"
Oct 30 22:20:29.982046 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:29.982027    5087 scope.go:122] "RemoveContainer" containerID="6dcd021446ef0739cb2c767c0feb2964e02fea0201b1aa82942e28c4494d864a"
Oct 30 22:20:29.990134 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:29.990114    5087 scope.go:122] "RemoveContainer" containerID="66f34129e1bbb0a909a0ef9e8d7700d7579f9bd5f90abd6e9d0487aca305f5bd"
Oct 30 22:20:43.237692 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:43.237603    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-20c7c7-j8jcp"]
Oct 30 22:20:43.286765 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:43.286735    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-20c7c7-j8jcp"]
Oct 30 22:20:43.708577 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:20:43.708499    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5" path="/var/lib/kubelet/pods/82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5/volumes"
Oct 30 22:20:47.170293 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:20:47.170253    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:21:05.934817 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:05.934771    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"]
Oct 30 22:21:05.983943 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:05.983908    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"]
Oct 30 22:21:11.876133 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:11.876087    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx"]
Oct 30 22:21:12.045292 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:12.045254    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-632149ee-hd66x"]
Oct 30 22:21:12.047662 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:12.047639    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-632149ee-hd66x"]
Oct 30 22:21:12.900186 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:12.900150    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"]
Oct 30 22:21:12.900732 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:12.900451    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4" podUID="55c06e30-cb56-47cb-ae02-fc44e936bc48" containerName="istio-proxy" containerID="cri-o://75d5db3719f7c73fb7eab8feaa5664fa801998f4a88b7fb716630c3684d04245" gracePeriod=30
Oct 30 22:21:12.902155 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:12.902127    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"]
Oct 30 22:21:12.902364 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:12.902328    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9" podUID="076fae6b-35bc-408e-aec8-e45316a2e9fa" containerName="istio-proxy" containerID="cri-o://852fe6e3f484f446ece2ce23111cb2653b54947cf8d9a48e09a0329990c163b9" gracePeriod=30
Oct 30 22:21:13.715320 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:13.715286    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9bc679e-17cc-4b56-908d-093182b7dd78" path="/var/lib/kubelet/pods/f9bc679e-17cc-4b56-908d-093182b7dd78/volumes"
Oct 30 22:21:16.509963 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:16.509932    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b137c5c5-v9bgf"]
Oct 30 22:21:16.510405 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:16.510370    5087 cpu_manager.go:414] "containerMap: removing container" podUID="82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5" containerName="revoke-keys"
Oct 30 22:21:16.510405 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:16.510382    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5" containerName="revoke-keys"
Oct 30 22:21:16.510494 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:16.510469    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="82bd9090-7caf-4bf0-a6b6-480cc4a3e2f5" containerName="revoke-keys"
Oct 30 22:21:16.516839 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:16.516814    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b137c5c5-v9bgf"
Oct 30 22:21:16.571870 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:16.571838    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b137c5c5-v9bgf"]
Oct 30 22:21:16.618479 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:16.618450    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:21:16.618685 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:16.618647    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lnxvw\""
Oct 30 22:21:16.699280 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:16.699226    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5pgb\" (UniqueName: \"kubernetes.io/projected/5a1e4c4d-e470-4c64-8f52-6f156d97c450-kube-api-access-j5pgb\") pod \"maas-api-revoke-sub-b137c5c5-v9bgf\" (UID: \"5a1e4c4d-e470-4c64-8f52-6f156d97c450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b137c5c5-v9bgf"
Oct 30 22:21:16.699511 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:16.699295    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a1e4c4d-e470-4c64-8f52-6f156d97c450-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b137c5c5-v9bgf\" (UID: \"5a1e4c4d-e470-4c64-8f52-6f156d97c450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b137c5c5-v9bgf"
Oct 30 22:21:16.699511 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:16.699443    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a1e4c4d-e470-4c64-8f52-6f156d97c450-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b137c5c5-v9bgf\" (UID: \"5a1e4c4d-e470-4c64-8f52-6f156d97c450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b137c5c5-v9bgf"
Oct 30 22:21:16.800702 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:16.800623    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5pgb\" (UniqueName: \"kubernetes.io/projected/5a1e4c4d-e470-4c64-8f52-6f156d97c450-kube-api-access-j5pgb\") pod \"maas-api-revoke-sub-b137c5c5-v9bgf\" (UID: \"5a1e4c4d-e470-4c64-8f52-6f156d97c450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b137c5c5-v9bgf"
Oct 30 22:21:16.800702 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:16.800678    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a1e4c4d-e470-4c64-8f52-6f156d97c450-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b137c5c5-v9bgf\" (UID: \"5a1e4c4d-e470-4c64-8f52-6f156d97c450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b137c5c5-v9bgf"
Oct 30 22:21:16.800918 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:16.800717    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a1e4c4d-e470-4c64-8f52-6f156d97c450-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b137c5c5-v9bgf\" (UID: \"5a1e4c4d-e470-4c64-8f52-6f156d97c450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b137c5c5-v9bgf"
Oct 30 22:21:16.801341 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:16.801318    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a1e4c4d-e470-4c64-8f52-6f156d97c450-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b137c5c5-v9bgf\" (UID: \"5a1e4c4d-e470-4c64-8f52-6f156d97c450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b137c5c5-v9bgf"
Oct 30 22:21:16.803272 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:16.803247    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a1e4c4d-e470-4c64-8f52-6f156d97c450-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b137c5c5-v9bgf\" (UID: \"5a1e4c4d-e470-4c64-8f52-6f156d97c450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b137c5c5-v9bgf"
Oct 30 22:21:16.857451 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:16.857421    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5pgb\" (UniqueName: \"kubernetes.io/projected/5a1e4c4d-e470-4c64-8f52-6f156d97c450-kube-api-access-j5pgb\") pod \"maas-api-revoke-sub-b137c5c5-v9bgf\" (UID: \"5a1e4c4d-e470-4c64-8f52-6f156d97c450\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b137c5c5-v9bgf"
Oct 30 22:21:16.877324 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:16.877290    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx" podUID="eab3f328-6507-45d0-abbe-0da0ae5fa74f" containerName="payload-processing" containerID="cri-o://94b18d820afc4bdbd2c6ba390932c4b8b6fd9c730714a56bc2c57bac7b781957" gracePeriod=25
Oct 30 22:21:17.130297 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:17.130223    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b137c5c5-v9bgf"
Oct 30 22:21:17.170051 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:21:17.169006    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:21:17.502126 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:17.502100    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b137c5c5-v9bgf"]
Oct 30 22:21:17.504512 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:21:17.504483    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a1e4c4d_e470_4c64_8f52_6f156d97c450.slice/crio-ba8867d99bbd14a93725c1fae989990a8e4150cad50378bef2a0fefc9239aed7 WatchSource:0}: Error finding container ba8867d99bbd14a93725c1fae989990a8e4150cad50378bef2a0fefc9239aed7: Status 404 returned error can't find the container with id ba8867d99bbd14a93725c1fae989990a8e4150cad50378bef2a0fefc9239aed7
Oct 30 22:21:17.531425 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:17.531392    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b137c5c5-v9bgf" event={"ID":"5a1e4c4d-e470-4c64-8f52-6f156d97c450","Type":"ContainerStarted","Data":"ba8867d99bbd14a93725c1fae989990a8e4150cad50378bef2a0fefc9239aed7"}
Oct 30 22:21:18.163294 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.163269    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:21:18.178051 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.178027    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:21:18.316105 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.316014    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/076fae6b-35bc-408e-aec8-e45316a2e9fa-istio-token\" (UniqueName: \"kubernetes.io/projected/076fae6b-35bc-408e-aec8-e45316a2e9fa-istio-token\") pod \"076fae6b-35bc-408e-aec8-e45316a2e9fa\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") "
Oct 30 22:21:18.316105 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.316085    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/076fae6b-35bc-408e-aec8-e45316a2e9fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076fae6b-35bc-408e-aec8-e45316a2e9fa-istiod-ca-cert\") pod \"076fae6b-35bc-408e-aec8-e45316a2e9fa\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") "
Oct 30 22:21:18.316397 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.316119    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-workload-certs\") pod \"076fae6b-35bc-408e-aec8-e45316a2e9fa\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") "
Oct 30 22:21:18.316397 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.316149    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-istio-envoy\") pod \"55c06e30-cb56-47cb-ae02-fc44e936bc48\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") "
Oct 30 22:21:18.316397 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.316178    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-workload-socket\") pod \"076fae6b-35bc-408e-aec8-e45316a2e9fa\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") "
Oct 30 22:21:18.316397 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.316210    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/076fae6b-35bc-408e-aec8-e45316a2e9fa-kube-api-access-gw5dn\" (UniqueName: \"kubernetes.io/projected/076fae6b-35bc-408e-aec8-e45316a2e9fa-kube-api-access-gw5dn\") pod \"076fae6b-35bc-408e-aec8-e45316a2e9fa\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") "
Oct 30 22:21:18.316397 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.316243    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55c06e30-cb56-47cb-ae02-fc44e936bc48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c06e30-cb56-47cb-ae02-fc44e936bc48-istiod-ca-cert\") pod \"55c06e30-cb56-47cb-ae02-fc44e936bc48\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") "
Oct 30 22:21:18.316397 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.316306    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-credential-socket\") pod \"076fae6b-35bc-408e-aec8-e45316a2e9fa\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") "
Oct 30 22:21:18.316397 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.316340    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55c06e30-cb56-47cb-ae02-fc44e936bc48-kube-api-access-dxdzv\" (UniqueName: \"kubernetes.io/projected/55c06e30-cb56-47cb-ae02-fc44e936bc48-kube-api-access-dxdzv\") pod \"55c06e30-cb56-47cb-ae02-fc44e936bc48\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") "
Oct 30 22:21:18.316845 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.316438    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/076fae6b-35bc-408e-aec8-e45316a2e9fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076fae6b-35bc-408e-aec8-e45316a2e9fa-istio-podinfo\") pod \"076fae6b-35bc-408e-aec8-e45316a2e9fa\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") "
Oct 30 22:21:18.316845 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.316487    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-istio-data\") pod \"55c06e30-cb56-47cb-ae02-fc44e936bc48\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") "
Oct 30 22:21:18.316845 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.316521    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-credential-socket\") pod \"55c06e30-cb56-47cb-ae02-fc44e936bc48\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") "
Oct 30 22:21:18.316845 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.316512    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-workload-socket" pod "076fae6b-35bc-408e-aec8-e45316a2e9fa" (UID: "076fae6b-35bc-408e-aec8-e45316a2e9fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:18.316845 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.316528    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-workload-certs" pod "076fae6b-35bc-408e-aec8-e45316a2e9fa" (UID: "076fae6b-35bc-408e-aec8-e45316a2e9fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:18.316845 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.316546    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/076fae6b-35bc-408e-aec8-e45316a2e9fa-istiod-ca-cert" pod "076fae6b-35bc-408e-aec8-e45316a2e9fa" (UID: "076fae6b-35bc-408e-aec8-e45316a2e9fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:18.316845 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.316641    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-credential-socket" pod "076fae6b-35bc-408e-aec8-e45316a2e9fa" (UID: "076fae6b-35bc-408e-aec8-e45316a2e9fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:18.316845 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.316695    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-workload-socket\") pod \"55c06e30-cb56-47cb-ae02-fc44e936bc48\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") "
Oct 30 22:21:18.316845 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.316725    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-istio-data\") pod \"076fae6b-35bc-408e-aec8-e45316a2e9fa\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") "
Oct 30 22:21:18.316845 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.316759    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-workload-certs\") pod \"55c06e30-cb56-47cb-ae02-fc44e936bc48\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") "
Oct 30 22:21:18.316845 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.316789    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55c06e30-cb56-47cb-ae02-fc44e936bc48-istio-token\" (UniqueName: \"kubernetes.io/projected/55c06e30-cb56-47cb-ae02-fc44e936bc48-istio-token\") pod \"55c06e30-cb56-47cb-ae02-fc44e936bc48\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") "
Oct 30 22:21:18.316845 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.316816    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-istio-envoy\") pod \"076fae6b-35bc-408e-aec8-e45316a2e9fa\" (UID: \"076fae6b-35bc-408e-aec8-e45316a2e9fa\") "
Oct 30 22:21:18.316845 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.316820    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-istio-data" pod "55c06e30-cb56-47cb-ae02-fc44e936bc48" (UID: "55c06e30-cb56-47cb-ae02-fc44e936bc48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:18.319198 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.316895    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/55c06e30-cb56-47cb-ae02-fc44e936bc48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c06e30-cb56-47cb-ae02-fc44e936bc48-istio-podinfo\") pod \"55c06e30-cb56-47cb-ae02-fc44e936bc48\" (UID: \"55c06e30-cb56-47cb-ae02-fc44e936bc48\") "
Oct 30 22:21:18.319198 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.316949    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-credential-socket" pod "55c06e30-cb56-47cb-ae02-fc44e936bc48" (UID: "55c06e30-cb56-47cb-ae02-fc44e936bc48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:18.319198 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.317047    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-istio-data" pod "076fae6b-35bc-408e-aec8-e45316a2e9fa" (UID: "076fae6b-35bc-408e-aec8-e45316a2e9fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:18.319198 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.317315    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/076fae6b-35bc-408e-aec8-e45316a2e9fa-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:18.319198 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.317337    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:18.319198 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.317372    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:18.319198 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.317389    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:18.319198 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.317404    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:18.319198 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.317417    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:18.319198 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.317434    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:18.319198 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.317478    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55c06e30-cb56-47cb-ae02-fc44e936bc48-istiod-ca-cert" pod "55c06e30-cb56-47cb-ae02-fc44e936bc48" (UID: "55c06e30-cb56-47cb-ae02-fc44e936bc48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:18.319198 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.317685    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-workload-socket" pod "55c06e30-cb56-47cb-ae02-fc44e936bc48" (UID: "55c06e30-cb56-47cb-ae02-fc44e936bc48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:18.319198 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.317814    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-workload-certs" pod "55c06e30-cb56-47cb-ae02-fc44e936bc48" (UID: "55c06e30-cb56-47cb-ae02-fc44e936bc48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:18.320165 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.319462    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/076fae6b-35bc-408e-aec8-e45316a2e9fa-istio-podinfo" pod "076fae6b-35bc-408e-aec8-e45316a2e9fa" (UID: "076fae6b-35bc-408e-aec8-e45316a2e9fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:21:18.320165 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.319636    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/076fae6b-35bc-408e-aec8-e45316a2e9fa-istio-token" pod "076fae6b-35bc-408e-aec8-e45316a2e9fa" (UID: "076fae6b-35bc-408e-aec8-e45316a2e9fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:18.320165 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.319933    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-istio-envoy" pod "55c06e30-cb56-47cb-ae02-fc44e936bc48" (UID: "55c06e30-cb56-47cb-ae02-fc44e936bc48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:18.320408 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.320210    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c06e30-cb56-47cb-ae02-fc44e936bc48-istio-token" pod "55c06e30-cb56-47cb-ae02-fc44e936bc48" (UID: "55c06e30-cb56-47cb-ae02-fc44e936bc48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:18.320408 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.320322    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/076fae6b-35bc-408e-aec8-e45316a2e9fa-kube-api-access-gw5dn" pod "076fae6b-35bc-408e-aec8-e45316a2e9fa" (UID: "076fae6b-35bc-408e-aec8-e45316a2e9fa"). InnerVolumeSpecName "kube-api-access-gw5dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:18.320683 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.320660    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-istio-envoy" pod "076fae6b-35bc-408e-aec8-e45316a2e9fa" (UID: "076fae6b-35bc-408e-aec8-e45316a2e9fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:18.320829 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.320812    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/55c06e30-cb56-47cb-ae02-fc44e936bc48-istio-podinfo" pod "55c06e30-cb56-47cb-ae02-fc44e936bc48" (UID: "55c06e30-cb56-47cb-ae02-fc44e936bc48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:21:18.321264 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.321230    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55c06e30-cb56-47cb-ae02-fc44e936bc48-kube-api-access-dxdzv" pod "55c06e30-cb56-47cb-ae02-fc44e936bc48" (UID: "55c06e30-cb56-47cb-ae02-fc44e936bc48"). InnerVolumeSpecName "kube-api-access-dxdzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:18.418768 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.418707    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:18.418884 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.418777    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw5dn\" (UniqueName: \"kubernetes.io/projected/076fae6b-35bc-408e-aec8-e45316a2e9fa-kube-api-access-gw5dn\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:18.418884 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.418795    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/55c06e30-cb56-47cb-ae02-fc44e936bc48-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:18.418884 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.418809    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxdzv\" (UniqueName: \"kubernetes.io/projected/55c06e30-cb56-47cb-ae02-fc44e936bc48-kube-api-access-dxdzv\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:18.418884 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.418824    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/076fae6b-35bc-408e-aec8-e45316a2e9fa-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:18.418884 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.418838    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:18.418884 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.418851    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/55c06e30-cb56-47cb-ae02-fc44e936bc48-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:18.418884 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.418865    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/55c06e30-cb56-47cb-ae02-fc44e936bc48-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:18.418884 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.418878    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/076fae6b-35bc-408e-aec8-e45316a2e9fa-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:18.419152 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.418892    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/55c06e30-cb56-47cb-ae02-fc44e936bc48-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:18.419152 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.418906    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/076fae6b-35bc-408e-aec8-e45316a2e9fa-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:18.538628 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.538591    5087 generic.go:356] "Generic (PLEG): container finished" podID="5a1e4c4d-e470-4c64-8f52-6f156d97c450" containerID="7cb0bb93c9c27b0f87aecf8fddb39d9cb1dfd0b97d1b40ad47f3c3006f2dd5f4" exitCode=0
Oct 30 22:21:18.539116 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.538670    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b137c5c5-v9bgf" event={"ID":"5a1e4c4d-e470-4c64-8f52-6f156d97c450","Type":"ContainerDied","Data":"7cb0bb93c9c27b0f87aecf8fddb39d9cb1dfd0b97d1b40ad47f3c3006f2dd5f4"}
Oct 30 22:21:18.540239 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.540214    5087 generic.go:356] "Generic (PLEG): container finished" podID="076fae6b-35bc-408e-aec8-e45316a2e9fa" containerID="852fe6e3f484f446ece2ce23111cb2653b54947cf8d9a48e09a0329990c163b9" exitCode=0
Oct 30 22:21:18.540406 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.540258    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"
Oct 30 22:21:18.540406 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.540299    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9" event={"ID":"076fae6b-35bc-408e-aec8-e45316a2e9fa","Type":"ContainerDied","Data":"852fe6e3f484f446ece2ce23111cb2653b54947cf8d9a48e09a0329990c163b9"}
Oct 30 22:21:18.540406 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.540384    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9" event={"ID":"076fae6b-35bc-408e-aec8-e45316a2e9fa","Type":"ContainerDied","Data":"b585cf964dff1b63c647fc25c1ccda39fba56c2fde4e31a301793165ed657537"}
Oct 30 22:21:18.540599 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.540419    5087 scope.go:122] "RemoveContainer" containerID="852fe6e3f484f446ece2ce23111cb2653b54947cf8d9a48e09a0329990c163b9"
Oct 30 22:21:18.541830 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.541799    5087 generic.go:356] "Generic (PLEG): container finished" podID="55c06e30-cb56-47cb-ae02-fc44e936bc48" containerID="75d5db3719f7c73fb7eab8feaa5664fa801998f4a88b7fb716630c3684d04245" exitCode=0
Oct 30 22:21:18.541929 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.541842    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"
Oct 30 22:21:18.541929 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.541859    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4" event={"ID":"55c06e30-cb56-47cb-ae02-fc44e936bc48","Type":"ContainerDied","Data":"75d5db3719f7c73fb7eab8feaa5664fa801998f4a88b7fb716630c3684d04245"}
Oct 30 22:21:18.541929 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.541902    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4" event={"ID":"55c06e30-cb56-47cb-ae02-fc44e936bc48","Type":"ContainerDied","Data":"95e9d0a989c284e56183106807939cc3c626a15a7f8233f7f3faf10b6856cf9e"}
Oct 30 22:21:18.551053 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.550982    5087 scope.go:122] "RemoveContainer" containerID="852fe6e3f484f446ece2ce23111cb2653b54947cf8d9a48e09a0329990c163b9"
Oct 30 22:21:18.551296 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:21:18.551278    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"852fe6e3f484f446ece2ce23111cb2653b54947cf8d9a48e09a0329990c163b9\": container with ID starting with 852fe6e3f484f446ece2ce23111cb2653b54947cf8d9a48e09a0329990c163b9 not found: ID does not exist" containerID="852fe6e3f484f446ece2ce23111cb2653b54947cf8d9a48e09a0329990c163b9"
Oct 30 22:21:18.551416 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.551305    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"852fe6e3f484f446ece2ce23111cb2653b54947cf8d9a48e09a0329990c163b9"} err="failed to get container status \"852fe6e3f484f446ece2ce23111cb2653b54947cf8d9a48e09a0329990c163b9\": rpc error: code = NotFound desc = could not find container \"852fe6e3f484f446ece2ce23111cb2653b54947cf8d9a48e09a0329990c163b9\": container with ID starting with 852fe6e3f484f446ece2ce23111cb2653b54947cf8d9a48e09a0329990c163b9 not found: ID does not exist"
Oct 30 22:21:18.551416 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.551323    5087 scope.go:122] "RemoveContainer" containerID="75d5db3719f7c73fb7eab8feaa5664fa801998f4a88b7fb716630c3684d04245"
Oct 30 22:21:18.562444 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.562427    5087 scope.go:122] "RemoveContainer" containerID="75d5db3719f7c73fb7eab8feaa5664fa801998f4a88b7fb716630c3684d04245"
Oct 30 22:21:18.562717 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:21:18.562695    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75d5db3719f7c73fb7eab8feaa5664fa801998f4a88b7fb716630c3684d04245\": container with ID starting with 75d5db3719f7c73fb7eab8feaa5664fa801998f4a88b7fb716630c3684d04245 not found: ID does not exist" containerID="75d5db3719f7c73fb7eab8feaa5664fa801998f4a88b7fb716630c3684d04245"
Oct 30 22:21:18.562778 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.562728    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75d5db3719f7c73fb7eab8feaa5664fa801998f4a88b7fb716630c3684d04245"} err="failed to get container status \"75d5db3719f7c73fb7eab8feaa5664fa801998f4a88b7fb716630c3684d04245\": rpc error: code = NotFound desc = could not find container \"75d5db3719f7c73fb7eab8feaa5664fa801998f4a88b7fb716630c3684d04245\": container with ID starting with 75d5db3719f7c73fb7eab8feaa5664fa801998f4a88b7fb716630c3684d04245 not found: ID does not exist"
Oct 30 22:21:18.761257 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.761221    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"]
Oct 30 22:21:18.765490 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.765449    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-44qk9"]
Oct 30 22:21:18.875672 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.875467    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"]
Oct 30 22:21:18.879117 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:18.879089    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pptz4"]
Oct 30 22:21:19.700136 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:19.700107    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b137c5c5-v9bgf"
Oct 30 22:21:19.709266 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:19.709231    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="076fae6b-35bc-408e-aec8-e45316a2e9fa" path="/var/lib/kubelet/pods/076fae6b-35bc-408e-aec8-e45316a2e9fa/volumes"
Oct 30 22:21:19.710002 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:19.709980    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55c06e30-cb56-47cb-ae02-fc44e936bc48" path="/var/lib/kubelet/pods/55c06e30-cb56-47cb-ae02-fc44e936bc48/volumes"
Oct 30 22:21:19.832322 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:19.832253    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a1e4c4d-e470-4c64-8f52-6f156d97c450-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a1e4c4d-e470-4c64-8f52-6f156d97c450-maas-api-openshift-ca\") pod \"5a1e4c4d-e470-4c64-8f52-6f156d97c450\" (UID: \"5a1e4c4d-e470-4c64-8f52-6f156d97c450\") "
Oct 30 22:21:19.832486 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:19.832337    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a1e4c4d-e470-4c64-8f52-6f156d97c450-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a1e4c4d-e470-4c64-8f52-6f156d97c450-maas-api-xks-ca\") pod \"5a1e4c4d-e470-4c64-8f52-6f156d97c450\" (UID: \"5a1e4c4d-e470-4c64-8f52-6f156d97c450\") "
Oct 30 22:21:19.832526 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:19.832503    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a1e4c4d-e470-4c64-8f52-6f156d97c450-kube-api-access-j5pgb\" (UniqueName: \"kubernetes.io/projected/5a1e4c4d-e470-4c64-8f52-6f156d97c450-kube-api-access-j5pgb\") pod \"5a1e4c4d-e470-4c64-8f52-6f156d97c450\" (UID: \"5a1e4c4d-e470-4c64-8f52-6f156d97c450\") "
Oct 30 22:21:19.832747 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:19.832718    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a1e4c4d-e470-4c64-8f52-6f156d97c450-maas-api-openshift-ca" pod "5a1e4c4d-e470-4c64-8f52-6f156d97c450" (UID: "5a1e4c4d-e470-4c64-8f52-6f156d97c450"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:19.833250 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:19.832878    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a1e4c4d-e470-4c64-8f52-6f156d97c450-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:19.834584 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:19.834559    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a1e4c4d-e470-4c64-8f52-6f156d97c450-maas-api-xks-ca" pod "5a1e4c4d-e470-4c64-8f52-6f156d97c450" (UID: "5a1e4c4d-e470-4c64-8f52-6f156d97c450"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:21:19.834727 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:19.834709    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a1e4c4d-e470-4c64-8f52-6f156d97c450-kube-api-access-j5pgb" pod "5a1e4c4d-e470-4c64-8f52-6f156d97c450" (UID: "5a1e4c4d-e470-4c64-8f52-6f156d97c450"). InnerVolumeSpecName "kube-api-access-j5pgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:19.933900 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:19.933862    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a1e4c4d-e470-4c64-8f52-6f156d97c450-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:19.933900 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:19.933893    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5pgb\" (UniqueName: \"kubernetes.io/projected/5a1e4c4d-e470-4c64-8f52-6f156d97c450-kube-api-access-j5pgb\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:20.557047 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:20.557016    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b137c5c5-v9bgf"
Oct 30 22:21:20.557235 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:20.557011    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b137c5c5-v9bgf" event={"ID":"5a1e4c4d-e470-4c64-8f52-6f156d97c450","Type":"ContainerDied","Data":"ba8867d99bbd14a93725c1fae989990a8e4150cad50378bef2a0fefc9239aed7"}
Oct 30 22:21:20.557235 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:20.557138    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba8867d99bbd14a93725c1fae989990a8e4150cad50378bef2a0fefc9239aed7"
Oct 30 22:21:20.955882 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:20.955798    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main" containerID="cri-o://fe059f22e865360e5a4bc2ff2e119338c8a36721ccb5c36404a629f473c71da0" gracePeriod=45
Oct 30 22:21:21.004423 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.004387    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main" containerID="cri-o://4ed24b27a2be1f12193774bb8c54ef2f9d5220c094b1436ecc99ad4e6025166a" gracePeriod=45
Oct 30 22:21:21.254359 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.254328    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 22:21:21.277583 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.277560    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 22:21:21.316215 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.316183    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx" podUID="eab3f328-6507-45d0-abbe-0da0ae5fa74f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:21:21.347441 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.347417    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-tmp-dir\") pod \"1ae4d8fa-1b3c-45dd-8461-edfcc9707043\" (UID: \"1ae4d8fa-1b3c-45dd-8461-edfcc9707043\") "
Oct 30 22:21:21.347558 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.347475    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-home\" (UniqueName: \"kubernetes.io/empty-dir/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-home\") pod \"1ae4d8fa-1b3c-45dd-8461-edfcc9707043\" (UID: \"1ae4d8fa-1b3c-45dd-8461-edfcc9707043\") "
Oct 30 22:21:21.347558 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.347521    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-dshm\") pod \"1ae4d8fa-1b3c-45dd-8461-edfcc9707043\" (UID: \"1ae4d8fa-1b3c-45dd-8461-edfcc9707043\") "
Oct 30 22:21:21.347642 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.347571    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-model-cache\") pod \"1ae4d8fa-1b3c-45dd-8461-edfcc9707043\" (UID: \"1ae4d8fa-1b3c-45dd-8461-edfcc9707043\") "
Oct 30 22:21:21.347642 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.347602    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-tls-certs\") pod \"1ae4d8fa-1b3c-45dd-8461-edfcc9707043\" (UID: \"1ae4d8fa-1b3c-45dd-8461-edfcc9707043\") "
Oct 30 22:21:21.347749 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.347643    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-tmp-dir" pod "1ae4d8fa-1b3c-45dd-8461-edfcc9707043" (UID: "1ae4d8fa-1b3c-45dd-8461-edfcc9707043"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:21.347749 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.347678    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-kube-api-access-76v2f\" (UniqueName: \"kubernetes.io/projected/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-kube-api-access-76v2f\") pod \"1ae4d8fa-1b3c-45dd-8461-edfcc9707043\" (UID: \"1ae4d8fa-1b3c-45dd-8461-edfcc9707043\") "
Oct 30 22:21:21.347866 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.347757    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-home" pod "1ae4d8fa-1b3c-45dd-8461-edfcc9707043" (UID: "1ae4d8fa-1b3c-45dd-8461-edfcc9707043"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:21.347919 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.347862    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-model-cache" pod "1ae4d8fa-1b3c-45dd-8461-edfcc9707043" (UID: "1ae4d8fa-1b3c-45dd-8461-edfcc9707043"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:21.348019 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.348001    5087 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-model-cache\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:21.348085 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.348048    5087 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-tmp-dir\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:21.348085 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.348065    5087 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-home\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:21.350006 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.349985    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-kube-api-access-76v2f" pod "1ae4d8fa-1b3c-45dd-8461-edfcc9707043" (UID: "1ae4d8fa-1b3c-45dd-8461-edfcc9707043"). InnerVolumeSpecName "kube-api-access-76v2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:21.350095 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.350014    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-dshm" pod "1ae4d8fa-1b3c-45dd-8461-edfcc9707043" (UID: "1ae4d8fa-1b3c-45dd-8461-edfcc9707043"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:21.350095 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.350028    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-tls-certs" pod "1ae4d8fa-1b3c-45dd-8461-edfcc9707043" (UID: "1ae4d8fa-1b3c-45dd-8461-edfcc9707043"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:21:21.448812 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.448784    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8606cde7-ffa5-48c3-941e-989ff637b916-tls-certs\" (UniqueName: \"kubernetes.io/secret/8606cde7-ffa5-48c3-941e-989ff637b916-tls-certs\") pod \"8606cde7-ffa5-48c3-941e-989ff637b916\" (UID: \"8606cde7-ffa5-48c3-941e-989ff637b916\") "
Oct 30 22:21:21.448989 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.448851    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8606cde7-ffa5-48c3-941e-989ff637b916-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8606cde7-ffa5-48c3-941e-989ff637b916-tmp-dir\") pod \"8606cde7-ffa5-48c3-941e-989ff637b916\" (UID: \"8606cde7-ffa5-48c3-941e-989ff637b916\") "
Oct 30 22:21:21.448989 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.448888    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8606cde7-ffa5-48c3-941e-989ff637b916-home\" (UniqueName: \"kubernetes.io/empty-dir/8606cde7-ffa5-48c3-941e-989ff637b916-home\") pod \"8606cde7-ffa5-48c3-941e-989ff637b916\" (UID: \"8606cde7-ffa5-48c3-941e-989ff637b916\") "
Oct 30 22:21:21.448989 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.448935    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8606cde7-ffa5-48c3-941e-989ff637b916-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8606cde7-ffa5-48c3-941e-989ff637b916-dshm\") pod \"8606cde7-ffa5-48c3-941e-989ff637b916\" (UID: \"8606cde7-ffa5-48c3-941e-989ff637b916\") "
Oct 30 22:21:21.449183 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.449000    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8606cde7-ffa5-48c3-941e-989ff637b916-kube-api-access-sqklm\" (UniqueName: \"kubernetes.io/projected/8606cde7-ffa5-48c3-941e-989ff637b916-kube-api-access-sqklm\") pod \"8606cde7-ffa5-48c3-941e-989ff637b916\" (UID: \"8606cde7-ffa5-48c3-941e-989ff637b916\") "
Oct 30 22:21:21.449183 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.449083    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8606cde7-ffa5-48c3-941e-989ff637b916-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8606cde7-ffa5-48c3-941e-989ff637b916-model-cache\") pod \"8606cde7-ffa5-48c3-941e-989ff637b916\" (UID: \"8606cde7-ffa5-48c3-941e-989ff637b916\") "
Oct 30 22:21:21.449183 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.449157    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8606cde7-ffa5-48c3-941e-989ff637b916-tmp-dir" pod "8606cde7-ffa5-48c3-941e-989ff637b916" (UID: "8606cde7-ffa5-48c3-941e-989ff637b916"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:21.449389 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.449224    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8606cde7-ffa5-48c3-941e-989ff637b916-home" pod "8606cde7-ffa5-48c3-941e-989ff637b916" (UID: "8606cde7-ffa5-48c3-941e-989ff637b916"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:21.449389 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.449334    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8606cde7-ffa5-48c3-941e-989ff637b916-model-cache" pod "8606cde7-ffa5-48c3-941e-989ff637b916" (UID: "8606cde7-ffa5-48c3-941e-989ff637b916"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:21.449521 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.449478    5087 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-dshm\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:21.449521 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.449502    5087 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-tls-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:21.449521 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.449516    5087 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8606cde7-ffa5-48c3-941e-989ff637b916-model-cache\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:21.449692 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.449531    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76v2f\" (UniqueName: \"kubernetes.io/projected/1ae4d8fa-1b3c-45dd-8461-edfcc9707043-kube-api-access-76v2f\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:21.449692 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.449549    5087 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8606cde7-ffa5-48c3-941e-989ff637b916-tmp-dir\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:21.449692 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.449562    5087 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8606cde7-ffa5-48c3-941e-989ff637b916-home\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:21.451804 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.451774    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8606cde7-ffa5-48c3-941e-989ff637b916-tls-certs" pod "8606cde7-ffa5-48c3-941e-989ff637b916" (UID: "8606cde7-ffa5-48c3-941e-989ff637b916"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:21:21.451912 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.451781    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8606cde7-ffa5-48c3-941e-989ff637b916-dshm" pod "8606cde7-ffa5-48c3-941e-989ff637b916" (UID: "8606cde7-ffa5-48c3-941e-989ff637b916"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:21:21.452022 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.451997    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8606cde7-ffa5-48c3-941e-989ff637b916-kube-api-access-sqklm" pod "8606cde7-ffa5-48c3-941e-989ff637b916" (UID: "8606cde7-ffa5-48c3-941e-989ff637b916"). InnerVolumeSpecName "kube-api-access-sqklm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:21.550649 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.550620    5087 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8606cde7-ffa5-48c3-941e-989ff637b916-dshm\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:21.550649 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.550648    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqklm\" (UniqueName: \"kubernetes.io/projected/8606cde7-ffa5-48c3-941e-989ff637b916-kube-api-access-sqklm\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:21.550649 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.550657    5087 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8606cde7-ffa5-48c3-941e-989ff637b916-tls-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:21.563584 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.563554    5087 generic.go:356] "Generic (PLEG): container finished" podID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerID="fe059f22e865360e5a4bc2ff2e119338c8a36721ccb5c36404a629f473c71da0" exitCode=0
Oct 30 22:21:21.563730 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.563609    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"
Oct 30 22:21:21.563730 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.563637    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" event={"ID":"1ae4d8fa-1b3c-45dd-8461-edfcc9707043","Type":"ContainerDied","Data":"fe059f22e865360e5a4bc2ff2e119338c8a36721ccb5c36404a629f473c71da0"}
Oct 30 22:21:21.563730 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.563680    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw" event={"ID":"1ae4d8fa-1b3c-45dd-8461-edfcc9707043","Type":"ContainerDied","Data":"445607c30fe69d33f7e6c5ab715d28138eaa12767a40288ac7cf5efa74e9d0ab"}
Oct 30 22:21:21.563730 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.563699    5087 scope.go:122] "RemoveContainer" containerID="fe059f22e865360e5a4bc2ff2e119338c8a36721ccb5c36404a629f473c71da0"
Oct 30 22:21:21.565104 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.565082    5087 generic.go:356] "Generic (PLEG): container finished" podID="8606cde7-ffa5-48c3-941e-989ff637b916" containerID="4ed24b27a2be1f12193774bb8c54ef2f9d5220c094b1436ecc99ad4e6025166a" exitCode=0
Oct 30 22:21:21.565238 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.565130    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" event={"ID":"8606cde7-ffa5-48c3-941e-989ff637b916","Type":"ContainerDied","Data":"4ed24b27a2be1f12193774bb8c54ef2f9d5220c094b1436ecc99ad4e6025166a"}
Oct 30 22:21:21.565238 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.565132    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"
Oct 30 22:21:21.565238 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.565152    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt" event={"ID":"8606cde7-ffa5-48c3-941e-989ff637b916","Type":"ContainerDied","Data":"11ea41fa6a29854dcdf6215e01dc3ca5bef9469918a45cd5b6086ee65b4e347a"}
Oct 30 22:21:21.573147 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.573124    5087 scope.go:122] "RemoveContainer" containerID="fe059f22e865360e5a4bc2ff2e119338c8a36721ccb5c36404a629f473c71da0"
Oct 30 22:21:21.573458 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:21:21.573429    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe059f22e865360e5a4bc2ff2e119338c8a36721ccb5c36404a629f473c71da0\": container with ID starting with fe059f22e865360e5a4bc2ff2e119338c8a36721ccb5c36404a629f473c71da0 not found: ID does not exist" containerID="fe059f22e865360e5a4bc2ff2e119338c8a36721ccb5c36404a629f473c71da0"
Oct 30 22:21:21.573559 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.573470    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe059f22e865360e5a4bc2ff2e119338c8a36721ccb5c36404a629f473c71da0"} err="failed to get container status \"fe059f22e865360e5a4bc2ff2e119338c8a36721ccb5c36404a629f473c71da0\": rpc error: code = NotFound desc = could not find container \"fe059f22e865360e5a4bc2ff2e119338c8a36721ccb5c36404a629f473c71da0\": container with ID starting with fe059f22e865360e5a4bc2ff2e119338c8a36721ccb5c36404a629f473c71da0 not found: ID does not exist"
Oct 30 22:21:21.573559 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.573496    5087 scope.go:122] "RemoveContainer" containerID="4ed24b27a2be1f12193774bb8c54ef2f9d5220c094b1436ecc99ad4e6025166a"
Oct 30 22:21:21.582189 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.582171    5087 scope.go:122] "RemoveContainer" containerID="4ed24b27a2be1f12193774bb8c54ef2f9d5220c094b1436ecc99ad4e6025166a"
Oct 30 22:21:21.582456 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:21:21.582437    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ed24b27a2be1f12193774bb8c54ef2f9d5220c094b1436ecc99ad4e6025166a\": container with ID starting with 4ed24b27a2be1f12193774bb8c54ef2f9d5220c094b1436ecc99ad4e6025166a not found: ID does not exist" containerID="4ed24b27a2be1f12193774bb8c54ef2f9d5220c094b1436ecc99ad4e6025166a"
Oct 30 22:21:21.582502 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.582464    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ed24b27a2be1f12193774bb8c54ef2f9d5220c094b1436ecc99ad4e6025166a"} err="failed to get container status \"4ed24b27a2be1f12193774bb8c54ef2f9d5220c094b1436ecc99ad4e6025166a\": rpc error: code = NotFound desc = could not find container \"4ed24b27a2be1f12193774bb8c54ef2f9d5220c094b1436ecc99ad4e6025166a\": container with ID starting with 4ed24b27a2be1f12193774bb8c54ef2f9d5220c094b1436ecc99ad4e6025166a not found: ID does not exist"
Oct 30 22:21:21.725200 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.725162    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"]
Oct 30 22:21:21.729263 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.729237    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-bc009f/premium-w3-bc009f-kserve-55c859665f-49gfw"]
Oct 30 22:21:21.840954 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.840881    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"]
Oct 30 22:21:21.844978 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:21.844954    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-bc009f/simulator-w3-bc009f-kserve-5d7667c858-q26tt"]
Oct 30 22:21:23.708754 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:23.708722    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" path="/var/lib/kubelet/pods/1ae4d8fa-1b3c-45dd-8461-edfcc9707043/volumes"
Oct 30 22:21:23.709182 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:23.709129    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" path="/var/lib/kubelet/pods/8606cde7-ffa5-48c3-941e-989ff637b916/volumes"
Oct 30 22:21:30.026393 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:30.026338    5087 scope.go:122] "RemoveContainer" containerID="702f157e34bc0576a404c4965ccaa2e506fe105c1ea877fe2b1c79f1914285e4"
Oct 30 22:21:31.317019 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:31.316977    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx" podUID="eab3f328-6507-45d0-abbe-0da0ae5fa74f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:21:37.136476 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:37.136386    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx"
Oct 30 22:21:37.192584 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:37.192550    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eab3f328-6507-45d0-abbe-0da0ae5fa74f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eab3f328-6507-45d0-abbe-0da0ae5fa74f-plugins-config-volume\") pod \"eab3f328-6507-45d0-abbe-0da0ae5fa74f\" (UID: \"eab3f328-6507-45d0-abbe-0da0ae5fa74f\") "
Oct 30 22:21:37.192584 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:37.192584    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eab3f328-6507-45d0-abbe-0da0ae5fa74f-kube-api-access-mmc7v\" (UniqueName: \"kubernetes.io/projected/eab3f328-6507-45d0-abbe-0da0ae5fa74f-kube-api-access-mmc7v\") pod \"eab3f328-6507-45d0-abbe-0da0ae5fa74f\" (UID: \"eab3f328-6507-45d0-abbe-0da0ae5fa74f\") "
Oct 30 22:21:37.192934 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:37.192909    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eab3f328-6507-45d0-abbe-0da0ae5fa74f-plugins-config-volume" pod "eab3f328-6507-45d0-abbe-0da0ae5fa74f" (UID: "eab3f328-6507-45d0-abbe-0da0ae5fa74f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:37.194979 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:37.194949    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eab3f328-6507-45d0-abbe-0da0ae5fa74f-kube-api-access-mmc7v" pod "eab3f328-6507-45d0-abbe-0da0ae5fa74f" (UID: "eab3f328-6507-45d0-abbe-0da0ae5fa74f"). InnerVolumeSpecName "kube-api-access-mmc7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:37.293812 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:37.293780    5087 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eab3f328-6507-45d0-abbe-0da0ae5fa74f-plugins-config-volume\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:37.293812 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:37.293805    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmc7v\" (UniqueName: \"kubernetes.io/projected/eab3f328-6507-45d0-abbe-0da0ae5fa74f-kube-api-access-mmc7v\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:21:37.656307 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:37.656267    5087 generic.go:356] "Generic (PLEG): container finished" podID="eab3f328-6507-45d0-abbe-0da0ae5fa74f" containerID="94b18d820afc4bdbd2c6ba390932c4b8b6fd9c730714a56bc2c57bac7b781957" exitCode=0
Oct 30 22:21:37.656525 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:37.656324    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx"
Oct 30 22:21:37.656525 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:37.656322    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx" event={"ID":"eab3f328-6507-45d0-abbe-0da0ae5fa74f","Type":"ContainerDied","Data":"94b18d820afc4bdbd2c6ba390932c4b8b6fd9c730714a56bc2c57bac7b781957"}
Oct 30 22:21:37.656525 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:37.656403    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx" event={"ID":"eab3f328-6507-45d0-abbe-0da0ae5fa74f","Type":"ContainerDied","Data":"78db747b591502dec16933611edb166b0dd6e238d2208366cf8a8f6ce0a19b11"}
Oct 30 22:21:37.656525 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:37.656435    5087 scope.go:122] "RemoveContainer" containerID="94b18d820afc4bdbd2c6ba390932c4b8b6fd9c730714a56bc2c57bac7b781957"
Oct 30 22:21:37.666208 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:37.666193    5087 scope.go:122] "RemoveContainer" containerID="94b18d820afc4bdbd2c6ba390932c4b8b6fd9c730714a56bc2c57bac7b781957"
Oct 30 22:21:37.666540 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:21:37.666521    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b18d820afc4bdbd2c6ba390932c4b8b6fd9c730714a56bc2c57bac7b781957\": container with ID starting with 94b18d820afc4bdbd2c6ba390932c4b8b6fd9c730714a56bc2c57bac7b781957 not found: ID does not exist" containerID="94b18d820afc4bdbd2c6ba390932c4b8b6fd9c730714a56bc2c57bac7b781957"
Oct 30 22:21:37.666624 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:37.666547    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b18d820afc4bdbd2c6ba390932c4b8b6fd9c730714a56bc2c57bac7b781957"} err="failed to get container status \"94b18d820afc4bdbd2c6ba390932c4b8b6fd9c730714a56bc2c57bac7b781957\": rpc error: code = NotFound desc = could not find container \"94b18d820afc4bdbd2c6ba390932c4b8b6fd9c730714a56bc2c57bac7b781957\": container with ID starting with 94b18d820afc4bdbd2c6ba390932c4b8b6fd9c730714a56bc2c57bac7b781957 not found: ID does not exist"
Oct 30 22:21:37.819544 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:37.819510    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx"]
Oct 30 22:21:37.823185 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:37.823160    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-bc009f-66fdc54f65-88jlx"]
Oct 30 22:21:39.707095 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:39.707062    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eab3f328-6507-45d0-abbe-0da0ae5fa74f" path="/var/lib/kubelet/pods/eab3f328-6507-45d0-abbe-0da0ae5fa74f/volumes"
Oct 30 22:21:47.172862 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:21:47.172817    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:21:50.053822 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:50.053787    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52bb6f1e-ndzqn"]
Oct 30 22:21:50.103580 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:50.103541    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52bb6f1e-ndzqn"]
Oct 30 22:21:51.708618 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:21:51.708580    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ace33c00-1c8a-4626-a72a-4d6e74a86a16" path="/var/lib/kubelet/pods/ace33c00-1c8a-4626-a72a-4d6e74a86a16/volumes"
Oct 30 22:22:13.405829 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.405746    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"]
Oct 30 22:22:13.406306 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.406134    5087 cpu_manager.go:414] "containerMap: removing container" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main"
Oct 30 22:22:13.406306 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.406144    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main"
Oct 30 22:22:13.406306 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.406158    5087 cpu_manager.go:414] "containerMap: removing container" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main"
Oct 30 22:22:13.406306 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.406163    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main"
Oct 30 22:22:13.406306 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.406173    5087 cpu_manager.go:414] "containerMap: removing container" podUID="5a1e4c4d-e470-4c64-8f52-6f156d97c450" containerName="revoke-keys"
Oct 30 22:22:13.406306 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.406179    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a1e4c4d-e470-4c64-8f52-6f156d97c450" containerName="revoke-keys"
Oct 30 22:22:13.406306 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.406193    5087 cpu_manager.go:414] "containerMap: removing container" podUID="55c06e30-cb56-47cb-ae02-fc44e936bc48" containerName="istio-proxy"
Oct 30 22:22:13.406306 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.406198    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55c06e30-cb56-47cb-ae02-fc44e936bc48" containerName="istio-proxy"
Oct 30 22:22:13.406306 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.406211    5087 cpu_manager.go:414] "containerMap: removing container" podUID="076fae6b-35bc-408e-aec8-e45316a2e9fa" containerName="istio-proxy"
Oct 30 22:22:13.406306 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.406217    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="076fae6b-35bc-408e-aec8-e45316a2e9fa" containerName="istio-proxy"
Oct 30 22:22:13.406306 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.406243    5087 cpu_manager.go:414] "containerMap: removing container" podUID="eab3f328-6507-45d0-abbe-0da0ae5fa74f" containerName="payload-processing"
Oct 30 22:22:13.406306 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.406248    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eab3f328-6507-45d0-abbe-0da0ae5fa74f" containerName="payload-processing"
Oct 30 22:22:13.406306 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.406305    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="8606cde7-ffa5-48c3-941e-989ff637b916" containerName="main"
Oct 30 22:22:13.406306 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.406312    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ae4d8fa-1b3c-45dd-8461-edfcc9707043" containerName="main"
Oct 30 22:22:13.406848 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.406320    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a1e4c4d-e470-4c64-8f52-6f156d97c450" containerName="revoke-keys"
Oct 30 22:22:13.406848 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.406328    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="55c06e30-cb56-47cb-ae02-fc44e936bc48" containerName="istio-proxy"
Oct 30 22:22:13.406848 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.406336    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="076fae6b-35bc-408e-aec8-e45316a2e9fa" containerName="istio-proxy"
Oct 30 22:22:13.406848 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.406342    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="eab3f328-6507-45d0-abbe-0da0ae5fa74f" containerName="payload-processing"
Oct 30 22:22:13.409695 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.409678    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:13.436606 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.436574    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-zbxls\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:13.436742 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.436686    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-zbxls\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:13.436742 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.436721    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fpbd\" (UniqueName: \"kubernetes.io/projected/84240bba-5ff2-4f87-8f9e-d5350089d27e-kube-api-access-4fpbd\") pod \"maas-default-gateway-openshift-default-74854d8ccf-zbxls\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:13.436838 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.436776    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84240bba-5ff2-4f87-8f9e-d5350089d27e-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-zbxls\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:13.436838 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.436804    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-zbxls\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:13.436966 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.436915    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-zbxls\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:13.437027 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.436977    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-zbxls\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:13.437027 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.437012    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84240bba-5ff2-4f87-8f9e-d5350089d27e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-zbxls\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:13.437118 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.437043    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84240bba-5ff2-4f87-8f9e-d5350089d27e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-zbxls\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:13.537975 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.537944    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-zbxls\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:13.538179 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.537989    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fpbd\" (UniqueName: \"kubernetes.io/projected/84240bba-5ff2-4f87-8f9e-d5350089d27e-kube-api-access-4fpbd\") pod \"maas-default-gateway-openshift-default-74854d8ccf-zbxls\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:13.538179 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.538101    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84240bba-5ff2-4f87-8f9e-d5350089d27e-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-zbxls\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:13.538179 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.538142    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-zbxls\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:13.538409 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.538190    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-zbxls\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:13.538409 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.538228    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-zbxls\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:13.538409 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.538254    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84240bba-5ff2-4f87-8f9e-d5350089d27e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-zbxls\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:13.538409 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.538283    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84240bba-5ff2-4f87-8f9e-d5350089d27e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-zbxls\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:13.538409 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.538385    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-zbxls\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:13.538712 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.538387    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-zbxls\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:13.538712 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.538554    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-zbxls\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:13.538712 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.538655    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-zbxls\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:13.538893 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.538799    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-zbxls\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:13.539022 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.539002    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84240bba-5ff2-4f87-8f9e-d5350089d27e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-zbxls\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:13.540798 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.540777    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-zbxls\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:13.541010 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.540993    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84240bba-5ff2-4f87-8f9e-d5350089d27e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-zbxls\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:13.595314 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.595281    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84240bba-5ff2-4f87-8f9e-d5350089d27e-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-zbxls\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:13.595491 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.595292    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fpbd\" (UniqueName: \"kubernetes.io/projected/84240bba-5ff2-4f87-8f9e-d5350089d27e-kube-api-access-4fpbd\") pod \"maas-default-gateway-openshift-default-74854d8ccf-zbxls\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:13.614681 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.614650    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"]
Oct 30 22:22:13.722266 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:13.722188    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:14.156302 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:14.156275    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"]
Oct 30 22:22:14.158438 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:22:14.158404    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84240bba_5ff2_4f87_8f9e_d5350089d27e.slice/crio-ac62d5d52fcf77ffa91a72516809123c61767416ebd3093b05bd3a046a581265 WatchSource:0}: Error finding container ac62d5d52fcf77ffa91a72516809123c61767416ebd3093b05bd3a046a581265: Status 404 returned error can't find the container with id ac62d5d52fcf77ffa91a72516809123c61767416ebd3093b05bd3a046a581265
Oct 30 22:22:14.160590 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:14.160558    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:22:14.160675 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:14.160625    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:22:14.160675 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:14.160657    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:22:14.844381 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:14.843891    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls" event={"ID":"84240bba-5ff2-4f87-8f9e-d5350089d27e","Type":"ContainerStarted","Data":"a381b1f224ca85ed41633366225d39686ee0c22f9b720ff970eaa2dd7e683c17"}
Oct 30 22:22:14.844381 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:14.843958    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls" event={"ID":"84240bba-5ff2-4f87-8f9e-d5350089d27e","Type":"ContainerStarted","Data":"ac62d5d52fcf77ffa91a72516809123c61767416ebd3093b05bd3a046a581265"}
Oct 30 22:22:15.012756 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:15.012705    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls" podStartSLOduration=2.012688494 podStartE2EDuration="2.012688494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:15.011985935 +0000 UTC m=+2929.486489914" watchObservedRunningTime="2026-10-30 22:22:15.012688494 +0000 UTC m=+2929.487192464"
Oct 30 22:22:15.723402 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:15.723360    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:15.724926 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:15.724900    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-zbxls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 22:22:15.725070 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:15.724951    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls" podUID="84240bba-5ff2-4f87-8f9e-d5350089d27e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 22:22:16.722693 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:16.722655    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-zbxls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 22:22:16.723093 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:16.722706    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls" podUID="84240bba-5ff2-4f87-8f9e-d5350089d27e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 22:22:17.182496 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:22:17.182394    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:22:17.723489 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:17.723402    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-zbxls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 22:22:17.723489 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:17.723458    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls" podUID="84240bba-5ff2-4f87-8f9e-d5350089d27e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 22:22:18.723327 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:18.723294    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-zbxls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 22:22:18.723556 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:18.723367    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls" podUID="84240bba-5ff2-4f87-8f9e-d5350089d27e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 22:22:19.723272 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:19.723237    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-zbxls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 22:22:19.723543 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:19.723289    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls" podUID="84240bba-5ff2-4f87-8f9e-d5350089d27e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 22:22:20.722920 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:20.722879    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-zbxls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 22:22:20.723402 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:20.722933    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls" podUID="84240bba-5ff2-4f87-8f9e-d5350089d27e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 22:22:21.722619 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:21.722586    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-zbxls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 30 22:22:21.722797 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:21.722658    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls" podUID="84240bba-5ff2-4f87-8f9e-d5350089d27e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 30 22:22:23.764564 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:23.764492    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-zbxls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:22:23.765026 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:23.764578    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls" podUID="84240bba-5ff2-4f87-8f9e-d5350089d27e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:22:24.693393 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:24.693331    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:24.693671 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:24.693650    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:24.694454 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:24.694434    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:22:28.589223 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.589192    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"]
Oct 30 22:22:28.592927 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.592906    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:28.659895 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.659860    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"]
Oct 30 22:22:28.694066 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.694036    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-v6jrr\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:28.694263 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.694079    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q449h\" (UniqueName: \"kubernetes.io/projected/41ec8a58-81a8-4586-8196-bb59d4bee8e4-kube-api-access-q449h\") pod \"maas-default-gateway-openshift-default-74854d8ccf-v6jrr\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:28.694263 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.694184    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-v6jrr\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:28.694263 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.694216    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-v6jrr\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:28.694263 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.694239    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-v6jrr\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:28.694547 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.694271    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-v6jrr\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:28.694547 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.694316    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-v6jrr\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:28.694547 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.694337    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-v6jrr\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:28.694547 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.694406    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-v6jrr\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:28.795649 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.795617    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-v6jrr\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:28.795649 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.795653    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-v6jrr\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:28.795934 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.795672    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-v6jrr\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:28.795934 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.795698    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-v6jrr\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:28.795934 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.795845    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-v6jrr\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:28.796108 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.795988    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-v6jrr\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:28.796108 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.796039    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q449h\" (UniqueName: \"kubernetes.io/projected/41ec8a58-81a8-4586-8196-bb59d4bee8e4-kube-api-access-q449h\") pod \"maas-default-gateway-openshift-default-74854d8ccf-v6jrr\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:28.796217 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.796146    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-v6jrr\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:28.796217 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.796149    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istio-data\") pod \"maas-default-gateway-openshift-default-74854d8ccf-v6jrr\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:28.796217 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.796181    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-v6jrr\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:28.796428 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.796245    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-credential-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-v6jrr\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:28.796562 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.796542    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-workload-certs\") pod \"maas-default-gateway-openshift-default-74854d8ccf-v6jrr\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:28.796630 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.796618    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-workload-socket\") pod \"maas-default-gateway-openshift-default-74854d8ccf-v6jrr\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:28.796856 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.796839    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-74854d8ccf-v6jrr\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:28.798696 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.798670    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istio-envoy\") pod \"maas-default-gateway-openshift-default-74854d8ccf-v6jrr\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:28.798859 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.798842    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-74854d8ccf-v6jrr\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:28.852406 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.852310    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istio-token\") pod \"maas-default-gateway-openshift-default-74854d8ccf-v6jrr\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:28.852579 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.852561    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q449h\" (UniqueName: \"kubernetes.io/projected/41ec8a58-81a8-4586-8196-bb59d4bee8e4-kube-api-access-q449h\") pod \"maas-default-gateway-openshift-default-74854d8ccf-v6jrr\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:28.905758 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:28.905728    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:29.330559 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:29.330534    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"]
Oct 30 22:22:29.332922 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:22:29.332896    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41ec8a58_81a8_4586_8196_bb59d4bee8e4.slice/crio-fc57968c78bb91ad65784ef2ed2d2c534693dd78bd8767950979fedf0f290c55 WatchSource:0}: Error finding container fc57968c78bb91ad65784ef2ed2d2c534693dd78bd8767950979fedf0f290c55: Status 404 returned error can't find the container with id fc57968c78bb91ad65784ef2ed2d2c534693dd78bd8767950979fedf0f290c55
Oct 30 22:22:29.335169 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:29.335138    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:22:29.335242 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:29.335201    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:22:29.335242 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:29.335228    5087 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 22:22:29.963445 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:29.963401    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr" event={"ID":"41ec8a58-81a8-4586-8196-bb59d4bee8e4","Type":"ContainerStarted","Data":"7ae8da5bad16087af3ff4a4af5c54d654242d5a1f7e13c227080ceebf57d0594"}
Oct 30 22:22:29.963885 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:29.963463    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr" event={"ID":"41ec8a58-81a8-4586-8196-bb59d4bee8e4","Type":"ContainerStarted","Data":"fc57968c78bb91ad65784ef2ed2d2c534693dd78bd8767950979fedf0f290c55"}
Oct 30 22:22:30.067741 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:30.067702    5087 scope.go:122] "RemoveContainer" containerID="f91d81855d20539814259143f047d9ffe76821db59e09ddd3d5093a255071487"
Oct 30 22:22:30.078304 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:30.078274    5087 scope.go:122] "RemoveContainer" containerID="6ace9120c54b06dc4e7b67fbff39dea29c5582d00e974a34f9c12b7d2560a199"
Oct 30 22:22:30.080999 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:30.080942    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr" podStartSLOduration=2.080925522 podStartE2EDuration="2.080925522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:30.079262251 +0000 UTC m=+2944.553766235" watchObservedRunningTime="2026-10-30 22:22:30.080925522 +0000 UTC m=+2944.555429502"
Oct 30 22:22:30.906547 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:30.906517    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:30.908150 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:30.908122    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-v6jrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 22:22:30.908304 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:30.908168    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr" podUID="41ec8a58-81a8-4586-8196-bb59d4bee8e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 22:22:31.906472 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:31.906441    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-v6jrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 22:22:31.906871 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:31.906490    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr" podUID="41ec8a58-81a8-4586-8196-bb59d4bee8e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 22:22:32.906662 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:32.906625    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-v6jrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 22:22:32.907082 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:32.906673    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr" podUID="41ec8a58-81a8-4586-8196-bb59d4bee8e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 22:22:33.906275 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:33.906239    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-v6jrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 22:22:33.906490 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:33.906295    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr" podUID="41ec8a58-81a8-4586-8196-bb59d4bee8e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 22:22:34.906099 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:34.906059    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-v6jrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 22:22:34.906549 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:34.906118    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr" podUID="41ec8a58-81a8-4586-8196-bb59d4bee8e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 22:22:35.906745 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:35.906698    5087 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-74854d8ccf-v6jrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body=
Oct 30 22:22:35.907282 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:35.906756    5087 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr" podUID="41ec8a58-81a8-4586-8196-bb59d4bee8e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused"
Oct 30 22:22:37.546384 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:37.546259    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:37.546930 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:37.546907    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:37.547756 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:22:37.547729    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:22:47.165928 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:22:47.165881    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:23:17.167330 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:23:17.167290    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:23:26.215958 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:23:26.215932    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 22:23:26.224600 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:23:26.224578    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 22:23:26.225230 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:23:26.225207    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 22:23:26.233533 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:23:26.233515    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 22:23:30.111782 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:23:30.111755    5087 scope.go:122] "RemoveContainer" containerID="a57e1f5dc2daf67b15ce94d642f109ad1ad5a123438ec2bacdc42d12bf35f94c"
Oct 30 22:23:47.174107 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:23:47.174027    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:24:17.167333 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:24:17.167291    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:24:47.166181 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:24:47.166141    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:25:17.166956 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:25:17.166875    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:25:30.158373 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:25:30.157756    5087 scope.go:122] "RemoveContainer" containerID="b3191aec0d7d315131b7726bd9f59a229ed694b1b6302e308271a638807f34e2"
Oct 30 22:25:47.166694 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:25:47.166660    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:26:17.166834 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:26:17.166787    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:26:20.061316 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:26:20.061274    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b137c5c5-v9bgf"]
Oct 30 22:26:20.109299 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:26:20.109268    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b137c5c5-v9bgf"]
Oct 30 22:26:21.707583 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:26:21.707543    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a1e4c4d-e470-4c64-8f52-6f156d97c450" path="/var/lib/kubelet/pods/5a1e4c4d-e470-4c64-8f52-6f156d97c450/volumes"
Oct 30 22:26:30.231737 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:26:30.231707    5087 scope.go:122] "RemoveContainer" containerID="84133266bc2f128e656785559d6e72ff02c81e9a5e56385876378213565aec9f"
Oct 30 22:26:30.240467 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:26:30.240446    5087 scope.go:122] "RemoveContainer" containerID="d739cf8766b04c6aedb4967d48be9eba87236a4e94df7b95c2476d50d3d55b0f"
Oct 30 22:26:47.167443 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:26:47.167332    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:27:17.165921 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:27:17.165885    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:27:30.266832 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:27:30.266802    5087 scope.go:122] "RemoveContainer" containerID="7cb0bb93c9c27b0f87aecf8fddb39d9cb1dfd0b97d1b40ad47f3c3006f2dd5f4"
Oct 30 22:27:47.166733 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:27:47.166699    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:28:17.167139 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:28:17.167058    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:28:26.248547 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:28:26.248513    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 22:28:26.256781 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:28:26.256752    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 22:28:26.257706 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:28:26.257688    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 22:28:26.266377 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:28:26.266359    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 22:28:47.167079 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:28:47.167042    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:29:17.167735 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:29:17.167694    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:29:47.167083 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:29:47.167004    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:30:17.167188 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:30:17.167151    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:30:47.167980 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:30:47.167940    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:31:17.167107 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:31:17.167025    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:31:47.168812 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:31:47.168774    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:32:17.166335 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:32:17.166300    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:32:37.172281 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:32:37.172198    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x4nzx"]
Oct 30 22:32:37.174732 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:32:37.172407    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x4nzx" podUID="385ea17a-3961-4253-b1b1-e8beaf700048" containerName="manager" containerID="cri-o://aca6bfc01649a5dd9d7cce880d747f697e6f2cda7f5a8a46416af9c1cf2d3699" gracePeriod=10
Oct 30 22:32:37.435294 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:32:37.435241    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x4nzx"
Oct 30 22:32:37.477230 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:32:37.477206    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/385ea17a-3961-4253-b1b1-e8beaf700048-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/385ea17a-3961-4253-b1b1-e8beaf700048-extensions-socket-volume\") pod \"385ea17a-3961-4253-b1b1-e8beaf700048\" (UID: \"385ea17a-3961-4253-b1b1-e8beaf700048\") "
Oct 30 22:32:37.477388 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:32:37.477244    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/385ea17a-3961-4253-b1b1-e8beaf700048-kube-api-access-kt7zb\" (UniqueName: \"kubernetes.io/projected/385ea17a-3961-4253-b1b1-e8beaf700048-kube-api-access-kt7zb\") pod \"385ea17a-3961-4253-b1b1-e8beaf700048\" (UID: \"385ea17a-3961-4253-b1b1-e8beaf700048\") "
Oct 30 22:32:37.477642 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:32:37.477611    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385ea17a-3961-4253-b1b1-e8beaf700048-extensions-socket-volume" pod "385ea17a-3961-4253-b1b1-e8beaf700048" (UID: "385ea17a-3961-4253-b1b1-e8beaf700048"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:32:37.479478 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:32:37.479456    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/385ea17a-3961-4253-b1b1-e8beaf700048-kube-api-access-kt7zb" pod "385ea17a-3961-4253-b1b1-e8beaf700048" (UID: "385ea17a-3961-4253-b1b1-e8beaf700048"). InnerVolumeSpecName "kube-api-access-kt7zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:32:37.578153 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:32:37.578116    5087 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/385ea17a-3961-4253-b1b1-e8beaf700048-extensions-socket-volume\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:37.578153 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:32:37.578156    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt7zb\" (UniqueName: \"kubernetes.io/projected/385ea17a-3961-4253-b1b1-e8beaf700048-kube-api-access-kt7zb\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:32:38.149258 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:32:38.149221    5087 generic.go:356] "Generic (PLEG): container finished" podID="385ea17a-3961-4253-b1b1-e8beaf700048" containerID="aca6bfc01649a5dd9d7cce880d747f697e6f2cda7f5a8a46416af9c1cf2d3699" exitCode=0
Oct 30 22:32:38.149483 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:32:38.149274    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x4nzx"
Oct 30 22:32:38.149483 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:32:38.149284    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x4nzx" event={"ID":"385ea17a-3961-4253-b1b1-e8beaf700048","Type":"ContainerDied","Data":"aca6bfc01649a5dd9d7cce880d747f697e6f2cda7f5a8a46416af9c1cf2d3699"}
Oct 30 22:32:38.149483 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:32:38.149338    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x4nzx" event={"ID":"385ea17a-3961-4253-b1b1-e8beaf700048","Type":"ContainerDied","Data":"4430758f7674a218b6c0b7e5d21b99686e0e2d48c03f1bf0dbc63ce14c126d56"}
Oct 30 22:32:38.149483 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:32:38.149387    5087 scope.go:122] "RemoveContainer" containerID="aca6bfc01649a5dd9d7cce880d747f697e6f2cda7f5a8a46416af9c1cf2d3699"
Oct 30 22:32:38.160829 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:32:38.160796    5087 scope.go:122] "RemoveContainer" containerID="aca6bfc01649a5dd9d7cce880d747f697e6f2cda7f5a8a46416af9c1cf2d3699"
Oct 30 22:32:38.161284 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:32:38.161265    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aca6bfc01649a5dd9d7cce880d747f697e6f2cda7f5a8a46416af9c1cf2d3699\": container with ID starting with aca6bfc01649a5dd9d7cce880d747f697e6f2cda7f5a8a46416af9c1cf2d3699 not found: ID does not exist" containerID="aca6bfc01649a5dd9d7cce880d747f697e6f2cda7f5a8a46416af9c1cf2d3699"
Oct 30 22:32:38.161417 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:32:38.161293    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aca6bfc01649a5dd9d7cce880d747f697e6f2cda7f5a8a46416af9c1cf2d3699"} err="failed to get container status \"aca6bfc01649a5dd9d7cce880d747f697e6f2cda7f5a8a46416af9c1cf2d3699\": rpc error: code = NotFound desc = could not find container \"aca6bfc01649a5dd9d7cce880d747f697e6f2cda7f5a8a46416af9c1cf2d3699\": container with ID starting with aca6bfc01649a5dd9d7cce880d747f697e6f2cda7f5a8a46416af9c1cf2d3699 not found: ID does not exist"
Oct 30 22:32:38.312037 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:32:38.312007    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x4nzx"]
Oct 30 22:32:38.361743 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:32:38.361714    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-x4nzx"]
Oct 30 22:32:39.707271 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:32:39.707242    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="385ea17a-3961-4253-b1b1-e8beaf700048" path="/var/lib/kubelet/pods/385ea17a-3961-4253-b1b1-e8beaf700048/volumes"
Oct 30 22:32:47.166991 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:32:47.166954    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:33:17.166154 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:33:17.166109    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:33:26.278712 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:33:26.278682    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 22:33:26.287082 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:33:26.287058    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 22:33:26.292223 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:33:26.292205    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 22:33:26.300863 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:33:26.300844    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 22:33:47.166672 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:33:47.166626    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:34:17.166742 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:34:17.166700    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:34:47.166702 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:34:47.166667    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:35:17.166621 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:35:17.166583    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:35:38.814192 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:38.814113    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"]
Oct 30 22:35:38.814896 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:38.814550    5087 cpu_manager.go:414] "containerMap: removing container" podUID="385ea17a-3961-4253-b1b1-e8beaf700048" containerName="manager"
Oct 30 22:35:38.814896 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:38.814565    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="385ea17a-3961-4253-b1b1-e8beaf700048" containerName="manager"
Oct 30 22:35:38.814896 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:38.814660    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="385ea17a-3961-4253-b1b1-e8beaf700048" containerName="manager"
Oct 30 22:35:38.818234 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:38.818213    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"
Oct 30 22:35:38.874773 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:38.874748    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-aafbca\"/\"default-dockercfg-pnkt9\""
Oct 30 22:35:38.924101 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:38.924071    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-aafbca\"/\"simulator-single-main-aafbca-kserve-self-signed-certs\""
Oct 30 22:35:38.924256 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:38.924181    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-aafbca\"/\"kube-root-ca.crt\""
Oct 30 22:35:38.924256 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:38.924246    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-aafbca\"/\"openshift-service-ca.crt\""
Oct 30 22:35:38.926066 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:38.926049    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"]
Oct 30 22:35:38.926405 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:38.926383    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-tmp-dir\") pod \"simulator-single-main-aafbca-kserve-694fb9fb4d-746rd\" (UID: \"ee7cdeea-57a4-411e-ac9a-fa255bb0113a\") " pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"
Oct 30 22:35:38.926504 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:38.926486    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbrkz\" (UniqueName: \"kubernetes.io/projected/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-kube-api-access-cbrkz\") pod \"simulator-single-main-aafbca-kserve-694fb9fb4d-746rd\" (UID: \"ee7cdeea-57a4-411e-ac9a-fa255bb0113a\") " pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"
Oct 30 22:35:38.926546 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:38.926530    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-model-cache\") pod \"simulator-single-main-aafbca-kserve-694fb9fb4d-746rd\" (UID: \"ee7cdeea-57a4-411e-ac9a-fa255bb0113a\") " pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"
Oct 30 22:35:38.926642 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:38.926623    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-home\") pod \"simulator-single-main-aafbca-kserve-694fb9fb4d-746rd\" (UID: \"ee7cdeea-57a4-411e-ac9a-fa255bb0113a\") " pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"
Oct 30 22:35:38.926729 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:38.926714    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-dshm\") pod \"simulator-single-main-aafbca-kserve-694fb9fb4d-746rd\" (UID: \"ee7cdeea-57a4-411e-ac9a-fa255bb0113a\") " pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"
Oct 30 22:35:38.926777 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:38.926760    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-tls-certs\") pod \"simulator-single-main-aafbca-kserve-694fb9fb4d-746rd\" (UID: \"ee7cdeea-57a4-411e-ac9a-fa255bb0113a\") " pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"
Oct 30 22:35:39.027704 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:39.027669    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-home\") pod \"simulator-single-main-aafbca-kserve-694fb9fb4d-746rd\" (UID: \"ee7cdeea-57a4-411e-ac9a-fa255bb0113a\") " pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"
Oct 30 22:35:39.027906 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:39.027778    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-dshm\") pod \"simulator-single-main-aafbca-kserve-694fb9fb4d-746rd\" (UID: \"ee7cdeea-57a4-411e-ac9a-fa255bb0113a\") " pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"
Oct 30 22:35:39.027906 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:39.027814    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-tls-certs\") pod \"simulator-single-main-aafbca-kserve-694fb9fb4d-746rd\" (UID: \"ee7cdeea-57a4-411e-ac9a-fa255bb0113a\") " pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"
Oct 30 22:35:39.027906 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:39.027865    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-tmp-dir\") pod \"simulator-single-main-aafbca-kserve-694fb9fb4d-746rd\" (UID: \"ee7cdeea-57a4-411e-ac9a-fa255bb0113a\") " pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"
Oct 30 22:35:39.028073 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:39.027906    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbrkz\" (UniqueName: \"kubernetes.io/projected/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-kube-api-access-cbrkz\") pod \"simulator-single-main-aafbca-kserve-694fb9fb4d-746rd\" (UID: \"ee7cdeea-57a4-411e-ac9a-fa255bb0113a\") " pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"
Oct 30 22:35:39.028073 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:39.027947    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-model-cache\") pod \"simulator-single-main-aafbca-kserve-694fb9fb4d-746rd\" (UID: \"ee7cdeea-57a4-411e-ac9a-fa255bb0113a\") " pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"
Oct 30 22:35:39.028189 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:39.028143    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-home\") pod \"simulator-single-main-aafbca-kserve-694fb9fb4d-746rd\" (UID: \"ee7cdeea-57a4-411e-ac9a-fa255bb0113a\") " pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"
Oct 30 22:35:39.028565 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:39.028510    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-model-cache\") pod \"simulator-single-main-aafbca-kserve-694fb9fb4d-746rd\" (UID: \"ee7cdeea-57a4-411e-ac9a-fa255bb0113a\") " pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"
Oct 30 22:35:39.028565 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:39.028538    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-tmp-dir\") pod \"simulator-single-main-aafbca-kserve-694fb9fb4d-746rd\" (UID: \"ee7cdeea-57a4-411e-ac9a-fa255bb0113a\") " pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"
Oct 30 22:35:39.030301 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:39.030279    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-dshm\") pod \"simulator-single-main-aafbca-kserve-694fb9fb4d-746rd\" (UID: \"ee7cdeea-57a4-411e-ac9a-fa255bb0113a\") " pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"
Oct 30 22:35:39.030682 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:39.030662    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-tls-certs\") pod \"simulator-single-main-aafbca-kserve-694fb9fb4d-746rd\" (UID: \"ee7cdeea-57a4-411e-ac9a-fa255bb0113a\") " pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"
Oct 30 22:35:39.086527 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:39.086467    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbrkz\" (UniqueName: \"kubernetes.io/projected/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-kube-api-access-cbrkz\") pod \"simulator-single-main-aafbca-kserve-694fb9fb4d-746rd\" (UID: \"ee7cdeea-57a4-411e-ac9a-fa255bb0113a\") " pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"
Oct 30 22:35:39.138914 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:39.138886    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"
Oct 30 22:35:39.514539 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:39.514505    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"]
Oct 30 22:35:39.518136 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:35:39.518105    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee7cdeea_57a4_411e_ac9a_fa255bb0113a.slice/crio-e6c3823d6a04ca2c760cdb507f2ffcf8413d1e3f41433ce95def0e90bec3d354 WatchSource:0}: Error finding container e6c3823d6a04ca2c760cdb507f2ffcf8413d1e3f41433ce95def0e90bec3d354: Status 404 returned error can't find the container with id e6c3823d6a04ca2c760cdb507f2ffcf8413d1e3f41433ce95def0e90bec3d354
Oct 30 22:35:40.041620 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:40.041587    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd" event={"ID":"ee7cdeea-57a4-411e-ac9a-fa255bb0113a","Type":"ContainerStarted","Data":"c933ee39d0a552ecf5c2c8f4e6643ee8f9f2c19f7b6cef1a5ab034aeedc8fd7b"}
Oct 30 22:35:40.041997 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:40.041626    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd" event={"ID":"ee7cdeea-57a4-411e-ac9a-fa255bb0113a","Type":"ContainerStarted","Data":"e6c3823d6a04ca2c760cdb507f2ffcf8413d1e3f41433ce95def0e90bec3d354"}
Oct 30 22:35:40.139897 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:40.139870    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"
Oct 30 22:35:40.156748 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:40.156699    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd" podStartSLOduration=2.156683235 podStartE2EDuration="2.156683235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:40.156290541 +0000 UTC m=+3734.630794520" watchObservedRunningTime="2026-10-30 22:35:40.156683235 +0000 UTC m=+3734.631187214"
Oct 30 22:35:47.166401 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:35:47.166365    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:35:49.139191 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:49.139152    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"
Oct 30 22:35:49.151484 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:49.151458    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"
Oct 30 22:35:50.100075 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:35:50.100043    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"
Oct 30 22:36:17.166835 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:36:17.166787    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:36:38.990265 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:38.990227    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"]
Oct 30 22:36:47.166384 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:36:47.166318    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:36:54.010520 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:54.010456    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd" podUID="ee7cdeea-57a4-411e-ac9a-fa255bb0113a" containerName="main" containerID="cri-o://c933ee39d0a552ecf5c2c8f4e6643ee8f9f2c19f7b6cef1a5ab034aeedc8fd7b" gracePeriod=45
Oct 30 22:36:54.089455 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:54.089423    5087 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd" podUID="ee7cdeea-57a4-411e-ac9a-fa255bb0113a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.221:8000/ready\": dial tcp 10.132.0.221:8000: connect: connection refused"
Oct 30 22:36:54.268122 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:54.268059    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"
Oct 30 22:36:54.333127 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:54.333094    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-model-cache\") pod \"ee7cdeea-57a4-411e-ac9a-fa255bb0113a\" (UID: \"ee7cdeea-57a4-411e-ac9a-fa255bb0113a\") "
Oct 30 22:36:54.333269 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:54.333136    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-home\" (UniqueName: \"kubernetes.io/empty-dir/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-home\") pod \"ee7cdeea-57a4-411e-ac9a-fa255bb0113a\" (UID: \"ee7cdeea-57a4-411e-ac9a-fa255bb0113a\") "
Oct 30 22:36:54.333269 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:54.333172    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-tls-certs\") pod \"ee7cdeea-57a4-411e-ac9a-fa255bb0113a\" (UID: \"ee7cdeea-57a4-411e-ac9a-fa255bb0113a\") "
Oct 30 22:36:54.333269 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:54.333203    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-kube-api-access-cbrkz\" (UniqueName: \"kubernetes.io/projected/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-kube-api-access-cbrkz\") pod \"ee7cdeea-57a4-411e-ac9a-fa255bb0113a\" (UID: \"ee7cdeea-57a4-411e-ac9a-fa255bb0113a\") "
Oct 30 22:36:54.333269 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:54.333223    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-dshm\") pod \"ee7cdeea-57a4-411e-ac9a-fa255bb0113a\" (UID: \"ee7cdeea-57a4-411e-ac9a-fa255bb0113a\") "
Oct 30 22:36:54.333495 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:54.333273    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-tmp-dir\") pod \"ee7cdeea-57a4-411e-ac9a-fa255bb0113a\" (UID: \"ee7cdeea-57a4-411e-ac9a-fa255bb0113a\") "
Oct 30 22:36:54.333495 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:54.333401    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-model-cache" pod "ee7cdeea-57a4-411e-ac9a-fa255bb0113a" (UID: "ee7cdeea-57a4-411e-ac9a-fa255bb0113a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:54.333618 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:54.333545    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-home" pod "ee7cdeea-57a4-411e-ac9a-fa255bb0113a" (UID: "ee7cdeea-57a4-411e-ac9a-fa255bb0113a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:54.333679 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:54.333627    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-tmp-dir" pod "ee7cdeea-57a4-411e-ac9a-fa255bb0113a" (UID: "ee7cdeea-57a4-411e-ac9a-fa255bb0113a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:54.333679 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:54.333668    5087 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-model-cache\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:54.333763 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:54.333687    5087 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-home\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:54.335599 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:54.335572    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-dshm" pod "ee7cdeea-57a4-411e-ac9a-fa255bb0113a" (UID: "ee7cdeea-57a4-411e-ac9a-fa255bb0113a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:54.335707 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:54.335617    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-tls-certs" pod "ee7cdeea-57a4-411e-ac9a-fa255bb0113a" (UID: "ee7cdeea-57a4-411e-ac9a-fa255bb0113a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:36:54.335707 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:54.335643    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-kube-api-access-cbrkz" pod "ee7cdeea-57a4-411e-ac9a-fa255bb0113a" (UID: "ee7cdeea-57a4-411e-ac9a-fa255bb0113a"). InnerVolumeSpecName "kube-api-access-cbrkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:54.406686 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:54.406655    5087 generic.go:356] "Generic (PLEG): container finished" podID="ee7cdeea-57a4-411e-ac9a-fa255bb0113a" containerID="c933ee39d0a552ecf5c2c8f4e6643ee8f9f2c19f7b6cef1a5ab034aeedc8fd7b" exitCode=0
Oct 30 22:36:54.406894 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:54.406727    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"
Oct 30 22:36:54.406894 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:54.406741    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd" event={"ID":"ee7cdeea-57a4-411e-ac9a-fa255bb0113a","Type":"ContainerDied","Data":"c933ee39d0a552ecf5c2c8f4e6643ee8f9f2c19f7b6cef1a5ab034aeedc8fd7b"}
Oct 30 22:36:54.406894 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:54.406808    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd" event={"ID":"ee7cdeea-57a4-411e-ac9a-fa255bb0113a","Type":"ContainerDied","Data":"e6c3823d6a04ca2c760cdb507f2ffcf8413d1e3f41433ce95def0e90bec3d354"}
Oct 30 22:36:54.406894 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:54.406832    5087 scope.go:122] "RemoveContainer" containerID="c933ee39d0a552ecf5c2c8f4e6643ee8f9f2c19f7b6cef1a5ab034aeedc8fd7b"
Oct 30 22:36:54.420427 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:54.420396    5087 scope.go:122] "RemoveContainer" containerID="c933ee39d0a552ecf5c2c8f4e6643ee8f9f2c19f7b6cef1a5ab034aeedc8fd7b"
Oct 30 22:36:54.420789 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:36:54.420760    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c933ee39d0a552ecf5c2c8f4e6643ee8f9f2c19f7b6cef1a5ab034aeedc8fd7b\": container with ID starting with c933ee39d0a552ecf5c2c8f4e6643ee8f9f2c19f7b6cef1a5ab034aeedc8fd7b not found: ID does not exist" containerID="c933ee39d0a552ecf5c2c8f4e6643ee8f9f2c19f7b6cef1a5ab034aeedc8fd7b"
Oct 30 22:36:54.420899 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:54.420801    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c933ee39d0a552ecf5c2c8f4e6643ee8f9f2c19f7b6cef1a5ab034aeedc8fd7b"} err="failed to get container status \"c933ee39d0a552ecf5c2c8f4e6643ee8f9f2c19f7b6cef1a5ab034aeedc8fd7b\": rpc error: code = NotFound desc = could not find container \"c933ee39d0a552ecf5c2c8f4e6643ee8f9f2c19f7b6cef1a5ab034aeedc8fd7b\": container with ID starting with c933ee39d0a552ecf5c2c8f4e6643ee8f9f2c19f7b6cef1a5ab034aeedc8fd7b not found: ID does not exist"
Oct 30 22:36:54.435155 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:54.435127    5087 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-tmp-dir\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:54.435155 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:54.435155    5087 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-tls-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:54.435321 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:54.435164    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbrkz\" (UniqueName: \"kubernetes.io/projected/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-kube-api-access-cbrkz\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:54.435321 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:54.435174    5087 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee7cdeea-57a4-411e-ac9a-fa255bb0113a-dshm\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:36:54.568546 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:54.568472    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"]
Oct 30 22:36:54.570039 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:54.570014    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-aafbca/simulator-single-main-aafbca-kserve-694fb9fb4d-746rd"]
Oct 30 22:36:55.709103 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:36:55.709074    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee7cdeea-57a4-411e-ac9a-fa255bb0113a" path="/var/lib/kubelet/pods/ee7cdeea-57a4-411e-ac9a-fa255bb0113a/volumes"
Oct 30 22:37:07.697983 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:07.697904    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"]
Oct 30 22:37:07.698495 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:07.698321    5087 cpu_manager.go:414] "containerMap: removing container" podUID="ee7cdeea-57a4-411e-ac9a-fa255bb0113a" containerName="main"
Oct 30 22:37:07.698495 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:07.698334    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee7cdeea-57a4-411e-ac9a-fa255bb0113a" containerName="main"
Oct 30 22:37:07.698495 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:07.698433    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee7cdeea-57a4-411e-ac9a-fa255bb0113a" containerName="main"
Oct 30 22:37:07.701672 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:07.701653    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"
Oct 30 22:37:07.802821 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:07.802790    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 22:37:07.803020 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:07.802978    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 22:37:07.803101 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:07.803032    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4xmws\""
Oct 30 22:37:07.803161 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:07.803117    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-12b4dd65-kserve-self-signed-certs\""
Oct 30 22:37:07.804826 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:07.804800    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"]
Oct 30 22:37:07.863076 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:07.863040    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-dshm\") pod \"e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h\" (UID: \"cf98ffa1-ddaf-460f-b783-9cf7340b3b14\") " pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"
Oct 30 22:37:07.863303 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:07.863086    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-model-cache\") pod \"e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h\" (UID: \"cf98ffa1-ddaf-460f-b783-9cf7340b3b14\") " pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"
Oct 30 22:37:07.863303 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:07.863107    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvbhj\" (UniqueName: \"kubernetes.io/projected/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-kube-api-access-xvbhj\") pod \"e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h\" (UID: \"cf98ffa1-ddaf-460f-b783-9cf7340b3b14\") " pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"
Oct 30 22:37:07.863303 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:07.863265    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-tls-certs\") pod \"e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h\" (UID: \"cf98ffa1-ddaf-460f-b783-9cf7340b3b14\") " pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"
Oct 30 22:37:07.863527 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:07.863313    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-home\") pod \"e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h\" (UID: \"cf98ffa1-ddaf-460f-b783-9cf7340b3b14\") " pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"
Oct 30 22:37:07.863527 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:07.863399    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-tmp-dir\") pod \"e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h\" (UID: \"cf98ffa1-ddaf-460f-b783-9cf7340b3b14\") " pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"
Oct 30 22:37:07.965049 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:07.964969    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-dshm\") pod \"e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h\" (UID: \"cf98ffa1-ddaf-460f-b783-9cf7340b3b14\") " pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"
Oct 30 22:37:07.965049 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:07.965012    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-model-cache\") pod \"e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h\" (UID: \"cf98ffa1-ddaf-460f-b783-9cf7340b3b14\") " pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"
Oct 30 22:37:07.965049 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:07.965036    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvbhj\" (UniqueName: \"kubernetes.io/projected/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-kube-api-access-xvbhj\") pod \"e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h\" (UID: \"cf98ffa1-ddaf-460f-b783-9cf7340b3b14\") " pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"
Oct 30 22:37:07.965331 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:07.965074    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-tls-certs\") pod \"e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h\" (UID: \"cf98ffa1-ddaf-460f-b783-9cf7340b3b14\") " pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"
Oct 30 22:37:07.965331 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:07.965099    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-home\") pod \"e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h\" (UID: \"cf98ffa1-ddaf-460f-b783-9cf7340b3b14\") " pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"
Oct 30 22:37:07.965331 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:07.965121    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-tmp-dir\") pod \"e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h\" (UID: \"cf98ffa1-ddaf-460f-b783-9cf7340b3b14\") " pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"
Oct 30 22:37:07.965839 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:07.965495    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-model-cache\") pod \"e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h\" (UID: \"cf98ffa1-ddaf-460f-b783-9cf7340b3b14\") " pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"
Oct 30 22:37:07.965839 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:07.965569    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-home\") pod \"e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h\" (UID: \"cf98ffa1-ddaf-460f-b783-9cf7340b3b14\") " pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"
Oct 30 22:37:07.965839 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:07.965656    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-tmp-dir\") pod \"e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h\" (UID: \"cf98ffa1-ddaf-460f-b783-9cf7340b3b14\") " pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"
Oct 30 22:37:07.967509 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:07.967490    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-dshm\") pod \"e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h\" (UID: \"cf98ffa1-ddaf-460f-b783-9cf7340b3b14\") " pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"
Oct 30 22:37:07.967874 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:07.967854    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-tls-certs\") pod \"e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h\" (UID: \"cf98ffa1-ddaf-460f-b783-9cf7340b3b14\") " pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"
Oct 30 22:37:08.023615 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:08.023579    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvbhj\" (UniqueName: \"kubernetes.io/projected/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-kube-api-access-xvbhj\") pod \"e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h\" (UID: \"cf98ffa1-ddaf-460f-b783-9cf7340b3b14\") " pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"
Oct 30 22:37:08.032215 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:08.032187    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"
Oct 30 22:37:08.418068 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:08.418039    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"]
Oct 30 22:37:08.420861 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:37:08.420834    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf98ffa1_ddaf_460f_b783_9cf7340b3b14.slice/crio-7ebb2caf4e9c747f75b42247cc8054ed91c082d8a1df62f9441f611d8647856e WatchSource:0}: Error finding container 7ebb2caf4e9c747f75b42247cc8054ed91c082d8a1df62f9441f611d8647856e: Status 404 returned error can't find the container with id 7ebb2caf4e9c747f75b42247cc8054ed91c082d8a1df62f9441f611d8647856e
Oct 30 22:37:08.476464 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:08.476425    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h" event={"ID":"cf98ffa1-ddaf-460f-b783-9cf7340b3b14","Type":"ContainerStarted","Data":"7ebb2caf4e9c747f75b42247cc8054ed91c082d8a1df62f9441f611d8647856e"}
Oct 30 22:37:09.482493 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:09.482464    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h" event={"ID":"cf98ffa1-ddaf-460f-b783-9cf7340b3b14","Type":"ContainerStarted","Data":"fb71f21151a01698be4c3a4ebcb2aeab1f9967e69ae435168aa007f3c73f98e0"}
Oct 30 22:37:09.597614 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:09.597563    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h" podStartSLOduration=2.597549148 podStartE2EDuration="2.597549148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:09.595930321 +0000 UTC m=+3824.070434297" watchObservedRunningTime="2026-10-30 22:37:09.597549148 +0000 UTC m=+3824.072053127"
Oct 30 22:37:10.032814 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:10.032774    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"
Oct 30 22:37:17.167981 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:37:17.167942    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:37:18.032602 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:18.032561    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"
Oct 30 22:37:18.045541 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:18.045510    5087 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"
Oct 30 22:37:18.543132 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:37:18.543104    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"
Oct 30 22:37:47.166894 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:37:47.166847    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:38:17.166332 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:38:17.166293    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:38:26.312049 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:26.312023    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 22:38:26.321141 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:26.321112    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 22:38:26.323249 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:26.323231    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 22:38:26.332255 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:26.332235    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 22:38:42.606879 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:42.606794    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"]
Oct 30 22:38:47.167791 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:38:47.167757    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:38:57.627450 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:57.627413    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h" podUID="cf98ffa1-ddaf-460f-b783-9cf7340b3b14" containerName="main" containerID="cri-o://fb71f21151a01698be4c3a4ebcb2aeab1f9967e69ae435168aa007f3c73f98e0" gracePeriod=45
Oct 30 22:38:57.891617 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:57.891560    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"
Oct 30 22:38:58.006926 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:58.006893    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-model-cache\") pod \"cf98ffa1-ddaf-460f-b783-9cf7340b3b14\" (UID: \"cf98ffa1-ddaf-460f-b783-9cf7340b3b14\") "
Oct 30 22:38:58.007090 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:58.006957    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-kube-api-access-xvbhj\" (UniqueName: \"kubernetes.io/projected/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-kube-api-access-xvbhj\") pod \"cf98ffa1-ddaf-460f-b783-9cf7340b3b14\" (UID: \"cf98ffa1-ddaf-460f-b783-9cf7340b3b14\") "
Oct 30 22:38:58.007090 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:58.006987    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-dshm\") pod \"cf98ffa1-ddaf-460f-b783-9cf7340b3b14\" (UID: \"cf98ffa1-ddaf-460f-b783-9cf7340b3b14\") "
Oct 30 22:38:58.007090 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:58.007056    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-tls-certs\") pod \"cf98ffa1-ddaf-460f-b783-9cf7340b3b14\" (UID: \"cf98ffa1-ddaf-460f-b783-9cf7340b3b14\") "
Oct 30 22:38:58.007215 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:58.007130    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-tmp-dir\") pod \"cf98ffa1-ddaf-460f-b783-9cf7340b3b14\" (UID: \"cf98ffa1-ddaf-460f-b783-9cf7340b3b14\") "
Oct 30 22:38:58.007215 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:58.007176    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-home\" (UniqueName: \"kubernetes.io/empty-dir/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-home\") pod \"cf98ffa1-ddaf-460f-b783-9cf7340b3b14\" (UID: \"cf98ffa1-ddaf-460f-b783-9cf7340b3b14\") "
Oct 30 22:38:58.007325 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:58.007174    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-model-cache" pod "cf98ffa1-ddaf-460f-b783-9cf7340b3b14" (UID: "cf98ffa1-ddaf-460f-b783-9cf7340b3b14"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:58.007483 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:58.007443    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-tmp-dir" pod "cf98ffa1-ddaf-460f-b783-9cf7340b3b14" (UID: "cf98ffa1-ddaf-460f-b783-9cf7340b3b14"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:58.007483 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:58.007452    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-home" pod "cf98ffa1-ddaf-460f-b783-9cf7340b3b14" (UID: "cf98ffa1-ddaf-460f-b783-9cf7340b3b14"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:58.007591 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:58.007572    5087 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-tmp-dir\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:58.007628 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:58.007590    5087 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-home\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:58.007628 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:58.007605    5087 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-model-cache\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:58.009716 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:58.009689    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-dshm" pod "cf98ffa1-ddaf-460f-b783-9cf7340b3b14" (UID: "cf98ffa1-ddaf-460f-b783-9cf7340b3b14"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:38:58.009823 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:58.009765    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-kube-api-access-xvbhj" pod "cf98ffa1-ddaf-460f-b783-9cf7340b3b14" (UID: "cf98ffa1-ddaf-460f-b783-9cf7340b3b14"). InnerVolumeSpecName "kube-api-access-xvbhj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:38:58.010821 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:58.010502    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-tls-certs" pod "cf98ffa1-ddaf-460f-b783-9cf7340b3b14" (UID: "cf98ffa1-ddaf-460f-b783-9cf7340b3b14"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:38:58.025092 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:58.025058    5087 generic.go:356] "Generic (PLEG): container finished" podID="cf98ffa1-ddaf-460f-b783-9cf7340b3b14" containerID="fb71f21151a01698be4c3a4ebcb2aeab1f9967e69ae435168aa007f3c73f98e0" exitCode=0
Oct 30 22:38:58.025221 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:58.025105    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"
Oct 30 22:38:58.025221 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:58.025131    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h" event={"ID":"cf98ffa1-ddaf-460f-b783-9cf7340b3b14","Type":"ContainerDied","Data":"fb71f21151a01698be4c3a4ebcb2aeab1f9967e69ae435168aa007f3c73f98e0"}
Oct 30 22:38:58.025221 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:58.025185    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h" event={"ID":"cf98ffa1-ddaf-460f-b783-9cf7340b3b14","Type":"ContainerDied","Data":"7ebb2caf4e9c747f75b42247cc8054ed91c082d8a1df62f9441f611d8647856e"}
Oct 30 22:38:58.025221 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:58.025206    5087 scope.go:122] "RemoveContainer" containerID="fb71f21151a01698be4c3a4ebcb2aeab1f9967e69ae435168aa007f3c73f98e0"
Oct 30 22:38:58.039205 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:58.039182    5087 scope.go:122] "RemoveContainer" containerID="fb71f21151a01698be4c3a4ebcb2aeab1f9967e69ae435168aa007f3c73f98e0"
Oct 30 22:38:58.039625 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:38:58.039600    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb71f21151a01698be4c3a4ebcb2aeab1f9967e69ae435168aa007f3c73f98e0\": container with ID starting with fb71f21151a01698be4c3a4ebcb2aeab1f9967e69ae435168aa007f3c73f98e0 not found: ID does not exist" containerID="fb71f21151a01698be4c3a4ebcb2aeab1f9967e69ae435168aa007f3c73f98e0"
Oct 30 22:38:58.039707 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:58.039638    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb71f21151a01698be4c3a4ebcb2aeab1f9967e69ae435168aa007f3c73f98e0"} err="failed to get container status \"fb71f21151a01698be4c3a4ebcb2aeab1f9967e69ae435168aa007f3c73f98e0\": rpc error: code = NotFound desc = could not find container \"fb71f21151a01698be4c3a4ebcb2aeab1f9967e69ae435168aa007f3c73f98e0\": container with ID starting with fb71f21151a01698be4c3a4ebcb2aeab1f9967e69ae435168aa007f3c73f98e0 not found: ID does not exist"
Oct 30 22:38:58.109022 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:58.109001    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvbhj\" (UniqueName: \"kubernetes.io/projected/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-kube-api-access-xvbhj\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:58.109022 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:58.109021    5087 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-dshm\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:58.109177 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:58.109030    5087 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf98ffa1-ddaf-460f-b783-9cf7340b3b14-tls-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:38:58.187103 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:58.187032    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"]
Oct 30 22:38:58.190694 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:58.190672    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-12b4dd65-kserve-557f9c8445-7cj2h"]
Oct 30 22:38:59.708209 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:38:59.708173    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf98ffa1-ddaf-460f-b783-9cf7340b3b14" path="/var/lib/kubelet/pods/cf98ffa1-ddaf-460f-b783-9cf7340b3b14/volumes"
Oct 30 22:39:17.166418 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:39:17.166332    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:39:47.166492 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:39:47.166454    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:40:05.162248 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:05.162168    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7975887845-4sw6w"]
Oct 30 22:40:05.162760 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:05.162624    5087 cpu_manager.go:414] "containerMap: removing container" podUID="cf98ffa1-ddaf-460f-b783-9cf7340b3b14" containerName="main"
Oct 30 22:40:05.162760 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:05.162642    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf98ffa1-ddaf-460f-b783-9cf7340b3b14" containerName="main"
Oct 30 22:40:05.162760 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:05.162725    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf98ffa1-ddaf-460f-b783-9cf7340b3b14" containerName="main"
Oct 30 22:40:05.165934 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:05.165913    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7975887845-4sw6w"
Oct 30 22:40:05.225139 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:05.225101    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52f12b96-99bb-40ef-85ca-7c9f691a1a3c-maas-api-tls\") pod \"maas-api-7975887845-4sw6w\" (UID: \"52f12b96-99bb-40ef-85ca-7c9f691a1a3c\") " pod="odh-ai-gateway-infra/maas-api-7975887845-4sw6w"
Oct 30 22:40:05.225387 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:05.225195    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52f12b96-99bb-40ef-85ca-7c9f691a1a3c-openshift-service-ca\") pod \"maas-api-7975887845-4sw6w\" (UID: \"52f12b96-99bb-40ef-85ca-7c9f691a1a3c\") " pod="odh-ai-gateway-infra/maas-api-7975887845-4sw6w"
Oct 30 22:40:05.225387 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:05.225238    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52f12b96-99bb-40ef-85ca-7c9f691a1a3c-metrics-certs\") pod \"maas-api-7975887845-4sw6w\" (UID: \"52f12b96-99bb-40ef-85ca-7c9f691a1a3c\") " pod="odh-ai-gateway-infra/maas-api-7975887845-4sw6w"
Oct 30 22:40:05.225387 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:05.225328    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts6k5\" (UniqueName: \"kubernetes.io/projected/52f12b96-99bb-40ef-85ca-7c9f691a1a3c-kube-api-access-ts6k5\") pod \"maas-api-7975887845-4sw6w\" (UID: \"52f12b96-99bb-40ef-85ca-7c9f691a1a3c\") " pod="odh-ai-gateway-infra/maas-api-7975887845-4sw6w"
Oct 30 22:40:05.269314 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:05.269191    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fqgfc\""
Oct 30 22:40:05.269314 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:05.269227    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 22:40:05.269314 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:05.269287    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 30 22:40:05.271275 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:05.271250    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7975887845-4sw6w"]
Oct 30 22:40:05.326800 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:05.326772    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52f12b96-99bb-40ef-85ca-7c9f691a1a3c-openshift-service-ca\") pod \"maas-api-7975887845-4sw6w\" (UID: \"52f12b96-99bb-40ef-85ca-7c9f691a1a3c\") " pod="odh-ai-gateway-infra/maas-api-7975887845-4sw6w"
Oct 30 22:40:05.326958 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:05.326815    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52f12b96-99bb-40ef-85ca-7c9f691a1a3c-metrics-certs\") pod \"maas-api-7975887845-4sw6w\" (UID: \"52f12b96-99bb-40ef-85ca-7c9f691a1a3c\") " pod="odh-ai-gateway-infra/maas-api-7975887845-4sw6w"
Oct 30 22:40:05.326958 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:05.326874    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts6k5\" (UniqueName: \"kubernetes.io/projected/52f12b96-99bb-40ef-85ca-7c9f691a1a3c-kube-api-access-ts6k5\") pod \"maas-api-7975887845-4sw6w\" (UID: \"52f12b96-99bb-40ef-85ca-7c9f691a1a3c\") " pod="odh-ai-gateway-infra/maas-api-7975887845-4sw6w"
Oct 30 22:40:05.326958 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:05.326894    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52f12b96-99bb-40ef-85ca-7c9f691a1a3c-maas-api-tls\") pod \"maas-api-7975887845-4sw6w\" (UID: \"52f12b96-99bb-40ef-85ca-7c9f691a1a3c\") " pod="odh-ai-gateway-infra/maas-api-7975887845-4sw6w"
Oct 30 22:40:05.327644 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:05.327585    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52f12b96-99bb-40ef-85ca-7c9f691a1a3c-openshift-service-ca\") pod \"maas-api-7975887845-4sw6w\" (UID: \"52f12b96-99bb-40ef-85ca-7c9f691a1a3c\") " pod="odh-ai-gateway-infra/maas-api-7975887845-4sw6w"
Oct 30 22:40:05.329476 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:05.329458    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52f12b96-99bb-40ef-85ca-7c9f691a1a3c-metrics-certs\") pod \"maas-api-7975887845-4sw6w\" (UID: \"52f12b96-99bb-40ef-85ca-7c9f691a1a3c\") " pod="odh-ai-gateway-infra/maas-api-7975887845-4sw6w"
Oct 30 22:40:05.329602 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:05.329584    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52f12b96-99bb-40ef-85ca-7c9f691a1a3c-maas-api-tls\") pod \"maas-api-7975887845-4sw6w\" (UID: \"52f12b96-99bb-40ef-85ca-7c9f691a1a3c\") " pod="odh-ai-gateway-infra/maas-api-7975887845-4sw6w"
Oct 30 22:40:05.384584 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:05.384550    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts6k5\" (UniqueName: \"kubernetes.io/projected/52f12b96-99bb-40ef-85ca-7c9f691a1a3c-kube-api-access-ts6k5\") pod \"maas-api-7975887845-4sw6w\" (UID: \"52f12b96-99bb-40ef-85ca-7c9f691a1a3c\") " pod="odh-ai-gateway-infra/maas-api-7975887845-4sw6w"
Oct 30 22:40:05.479599 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:05.479513    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7975887845-4sw6w"
Oct 30 22:40:05.852097 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:05.852051    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7975887845-4sw6w"]
Oct 30 22:40:05.854559 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:40:05.854532    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52f12b96_99bb_40ef_85ca_7c9f691a1a3c.slice/crio-54e3b12b62e05433ef55faa454acf693a167727f567f2c2fcf65770bb320185c WatchSource:0}: Error finding container 54e3b12b62e05433ef55faa454acf693a167727f567f2c2fcf65770bb320185c: Status 404 returned error can't find the container with id 54e3b12b62e05433ef55faa454acf693a167727f567f2c2fcf65770bb320185c
Oct 30 22:40:05.855885 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:05.855868    5087 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:40:06.377203 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:06.377168    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7975887845-4sw6w" event={"ID":"52f12b96-99bb-40ef-85ca-7c9f691a1a3c","Type":"ContainerStarted","Data":"54e3b12b62e05433ef55faa454acf693a167727f567f2c2fcf65770bb320185c"}
Oct 30 22:40:07.383976 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:07.383941    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7975887845-4sw6w" event={"ID":"52f12b96-99bb-40ef-85ca-7c9f691a1a3c","Type":"ContainerStarted","Data":"f6e981aedb261d2250bbecd8c78d6f01cb6f6c4318c389001f9debafb619bdcb"}
Oct 30 22:40:07.384439 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:07.384049    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7975887845-4sw6w"
Oct 30 22:40:07.497735 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:07.497674    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7975887845-4sw6w" podStartSLOduration=1.583188806 podStartE2EDuration="2.497660723s" totalImagesPullingTime="914.471917ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:07.497033336 +0000 UTC m=+4001.971537317" watchObservedRunningTime="2026-10-30 22:40:07.497660723 +0000 UTC m=+4001.972164702"
Oct 30 22:40:13.394609 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:13.394581    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7975887845-4sw6w"
Oct 30 22:40:13.511356 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:13.511315    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7975887845-4sw6w"]
Oct 30 22:40:13.511523 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:13.511500    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7975887845-4sw6w" podUID="52f12b96-99bb-40ef-85ca-7c9f691a1a3c" containerName="maas-api" containerID="cri-o://f6e981aedb261d2250bbecd8c78d6f01cb6f6c4318c389001f9debafb619bdcb" gracePeriod=30
Oct 30 22:40:13.774667 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:13.774640    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7975887845-4sw6w"
Oct 30 22:40:13.812828 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:13.812786    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52f12b96-99bb-40ef-85ca-7c9f691a1a3c-kube-api-access-ts6k5\" (UniqueName: \"kubernetes.io/projected/52f12b96-99bb-40ef-85ca-7c9f691a1a3c-kube-api-access-ts6k5\") pod \"52f12b96-99bb-40ef-85ca-7c9f691a1a3c\" (UID: \"52f12b96-99bb-40ef-85ca-7c9f691a1a3c\") "
Oct 30 22:40:13.813005 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:13.812895    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52f12b96-99bb-40ef-85ca-7c9f691a1a3c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/52f12b96-99bb-40ef-85ca-7c9f691a1a3c-metrics-certs\") pod \"52f12b96-99bb-40ef-85ca-7c9f691a1a3c\" (UID: \"52f12b96-99bb-40ef-85ca-7c9f691a1a3c\") "
Oct 30 22:40:13.813005 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:13.812931    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52f12b96-99bb-40ef-85ca-7c9f691a1a3c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52f12b96-99bb-40ef-85ca-7c9f691a1a3c-maas-api-tls\") pod \"52f12b96-99bb-40ef-85ca-7c9f691a1a3c\" (UID: \"52f12b96-99bb-40ef-85ca-7c9f691a1a3c\") "
Oct 30 22:40:13.813005 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:13.812993    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52f12b96-99bb-40ef-85ca-7c9f691a1a3c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52f12b96-99bb-40ef-85ca-7c9f691a1a3c-openshift-service-ca\") pod \"52f12b96-99bb-40ef-85ca-7c9f691a1a3c\" (UID: \"52f12b96-99bb-40ef-85ca-7c9f691a1a3c\") "
Oct 30 22:40:13.813460 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:13.813433    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52f12b96-99bb-40ef-85ca-7c9f691a1a3c-openshift-service-ca" pod "52f12b96-99bb-40ef-85ca-7c9f691a1a3c" (UID: "52f12b96-99bb-40ef-85ca-7c9f691a1a3c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:13.815586 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:13.815557    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52f12b96-99bb-40ef-85ca-7c9f691a1a3c-metrics-certs" pod "52f12b96-99bb-40ef-85ca-7c9f691a1a3c" (UID: "52f12b96-99bb-40ef-85ca-7c9f691a1a3c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:13.815754 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:13.815737    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52f12b96-99bb-40ef-85ca-7c9f691a1a3c-kube-api-access-ts6k5" pod "52f12b96-99bb-40ef-85ca-7c9f691a1a3c" (UID: "52f12b96-99bb-40ef-85ca-7c9f691a1a3c"). InnerVolumeSpecName "kube-api-access-ts6k5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:13.815822 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:13.815786    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52f12b96-99bb-40ef-85ca-7c9f691a1a3c-maas-api-tls" pod "52f12b96-99bb-40ef-85ca-7c9f691a1a3c" (UID: "52f12b96-99bb-40ef-85ca-7c9f691a1a3c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:13.914218 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:13.914185    5087 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52f12b96-99bb-40ef-85ca-7c9f691a1a3c-metrics-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.914218 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:13.914214    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52f12b96-99bb-40ef-85ca-7c9f691a1a3c-maas-api-tls\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.914218 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:13.914225    5087 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52f12b96-99bb-40ef-85ca-7c9f691a1a3c-openshift-service-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:13.914479 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:13.914234    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts6k5\" (UniqueName: \"kubernetes.io/projected/52f12b96-99bb-40ef-85ca-7c9f691a1a3c-kube-api-access-ts6k5\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:40:14.418853 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:14.418812    5087 generic.go:356] "Generic (PLEG): container finished" podID="52f12b96-99bb-40ef-85ca-7c9f691a1a3c" containerID="f6e981aedb261d2250bbecd8c78d6f01cb6f6c4318c389001f9debafb619bdcb" exitCode=0
Oct 30 22:40:14.419418 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:14.418888    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7975887845-4sw6w"
Oct 30 22:40:14.419418 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:14.418895    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7975887845-4sw6w" event={"ID":"52f12b96-99bb-40ef-85ca-7c9f691a1a3c","Type":"ContainerDied","Data":"f6e981aedb261d2250bbecd8c78d6f01cb6f6c4318c389001f9debafb619bdcb"}
Oct 30 22:40:14.419418 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:14.418996    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7975887845-4sw6w" event={"ID":"52f12b96-99bb-40ef-85ca-7c9f691a1a3c","Type":"ContainerDied","Data":"54e3b12b62e05433ef55faa454acf693a167727f567f2c2fcf65770bb320185c"}
Oct 30 22:40:14.419418 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:14.419032    5087 scope.go:122] "RemoveContainer" containerID="f6e981aedb261d2250bbecd8c78d6f01cb6f6c4318c389001f9debafb619bdcb"
Oct 30 22:40:14.428600 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:14.428580    5087 scope.go:122] "RemoveContainer" containerID="f6e981aedb261d2250bbecd8c78d6f01cb6f6c4318c389001f9debafb619bdcb"
Oct 30 22:40:14.428876 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:40:14.428855    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6e981aedb261d2250bbecd8c78d6f01cb6f6c4318c389001f9debafb619bdcb\": container with ID starting with f6e981aedb261d2250bbecd8c78d6f01cb6f6c4318c389001f9debafb619bdcb not found: ID does not exist" containerID="f6e981aedb261d2250bbecd8c78d6f01cb6f6c4318c389001f9debafb619bdcb"
Oct 30 22:40:14.428957 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:14.428883    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6e981aedb261d2250bbecd8c78d6f01cb6f6c4318c389001f9debafb619bdcb"} err="failed to get container status \"f6e981aedb261d2250bbecd8c78d6f01cb6f6c4318c389001f9debafb619bdcb\": rpc error: code = NotFound desc = could not find container \"f6e981aedb261d2250bbecd8c78d6f01cb6f6c4318c389001f9debafb619bdcb\": container with ID starting with f6e981aedb261d2250bbecd8c78d6f01cb6f6c4318c389001f9debafb619bdcb not found: ID does not exist"
Oct 30 22:40:14.580465 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:14.580427    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7975887845-4sw6w"]
Oct 30 22:40:14.629888 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:14.629854    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7975887845-4sw6w"]
Oct 30 22:40:15.707667 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:15.707630    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52f12b96-99bb-40ef-85ca-7c9f691a1a3c" path="/var/lib/kubelet/pods/52f12b96-99bb-40ef-85ca-7c9f691a1a3c/volumes"
Oct 30 22:40:17.166375 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:40:17.166308    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:40:19.503705 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:19.503670    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5ff4cfa5-7bdc4f6fb7-ktnhr"]
Oct 30 22:40:19.504296 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:19.504276    5087 cpu_manager.go:414] "containerMap: removing container" podUID="52f12b96-99bb-40ef-85ca-7c9f691a1a3c" containerName="maas-api"
Oct 30 22:40:19.504410 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:19.504299    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52f12b96-99bb-40ef-85ca-7c9f691a1a3c" containerName="maas-api"
Oct 30 22:40:19.504467 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:19.504421    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="52f12b96-99bb-40ef-85ca-7c9f691a1a3c" containerName="maas-api"
Oct 30 22:40:19.508731 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:19.508708    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ff4cfa5-7bdc4f6fb7-ktnhr"
Oct 30 22:40:19.566210 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:19.566126    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5ff4cfa5-7bdc4f6fb7-ktnhr"]
Oct 30 22:40:19.567179 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:19.567087    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvv8k\" (UniqueName: \"kubernetes.io/projected/c0da68f3-69e5-4825-ac6b-f62ade01fe68-kube-api-access-bvv8k\") pod \"payload-pre-processing-e2e-ait-5ff4cfa5-7bdc4f6fb7-ktnhr\" (UID: \"c0da68f3-69e5-4825-ac6b-f62ade01fe68\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5ff4cfa5-7bdc4f6fb7-ktnhr"
Oct 30 22:40:19.567508 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:19.567466    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0da68f3-69e5-4825-ac6b-f62ade01fe68-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5ff4cfa5-7bdc4f6fb7-ktnhr\" (UID: \"c0da68f3-69e5-4825-ac6b-f62ade01fe68\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5ff4cfa5-7bdc4f6fb7-ktnhr"
Oct 30 22:40:19.612584 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:19.612552    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5ff4cfa5\""
Oct 30 22:40:19.612584 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:19.612565    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5ff4cfa5-dockercfg-v9mr8\""
Oct 30 22:40:19.670453 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:19.670413    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvv8k\" (UniqueName: \"kubernetes.io/projected/c0da68f3-69e5-4825-ac6b-f62ade01fe68-kube-api-access-bvv8k\") pod \"payload-pre-processing-e2e-ait-5ff4cfa5-7bdc4f6fb7-ktnhr\" (UID: \"c0da68f3-69e5-4825-ac6b-f62ade01fe68\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5ff4cfa5-7bdc4f6fb7-ktnhr"
Oct 30 22:40:19.670654 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:19.670527    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0da68f3-69e5-4825-ac6b-f62ade01fe68-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5ff4cfa5-7bdc4f6fb7-ktnhr\" (UID: \"c0da68f3-69e5-4825-ac6b-f62ade01fe68\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5ff4cfa5-7bdc4f6fb7-ktnhr"
Oct 30 22:40:19.671245 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:19.671223    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0da68f3-69e5-4825-ac6b-f62ade01fe68-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5ff4cfa5-7bdc4f6fb7-ktnhr\" (UID: \"c0da68f3-69e5-4825-ac6b-f62ade01fe68\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5ff4cfa5-7bdc4f6fb7-ktnhr"
Oct 30 22:40:19.671958 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:19.671936    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5ff4cfa5-97fcb7787-4dfjj"]
Oct 30 22:40:19.675702 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:19.675684    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5ff4cfa5-97fcb7787-4dfjj"
Oct 30 22:40:19.710926 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:19.710892    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5ff4cfa5-97fcb7787-4dfjj"]
Oct 30 22:40:19.727068 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:19.727042    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvv8k\" (UniqueName: \"kubernetes.io/projected/c0da68f3-69e5-4825-ac6b-f62ade01fe68-kube-api-access-bvv8k\") pod \"payload-pre-processing-e2e-ait-5ff4cfa5-7bdc4f6fb7-ktnhr\" (UID: \"c0da68f3-69e5-4825-ac6b-f62ade01fe68\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5ff4cfa5-7bdc4f6fb7-ktnhr"
Oct 30 22:40:19.771205 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:19.771130    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4854a56-ebca-4747-9e54-a37e5a5eca75-plugins-config-volume\") pod \"payload-processing-e2e-ait-5ff4cfa5-97fcb7787-4dfjj\" (UID: \"a4854a56-ebca-4747-9e54-a37e5a5eca75\") " pod="openshift-ingress/payload-processing-e2e-ait-5ff4cfa5-97fcb7787-4dfjj"
Oct 30 22:40:19.771334 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:19.771310    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klz8r\" (UniqueName: \"kubernetes.io/projected/a4854a56-ebca-4747-9e54-a37e5a5eca75-kube-api-access-klz8r\") pod \"payload-processing-e2e-ait-5ff4cfa5-97fcb7787-4dfjj\" (UID: \"a4854a56-ebca-4747-9e54-a37e5a5eca75\") " pod="openshift-ingress/payload-processing-e2e-ait-5ff4cfa5-97fcb7787-4dfjj"
Oct 30 22:40:19.821689 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:19.821666    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ff4cfa5-7bdc4f6fb7-ktnhr"
Oct 30 22:40:19.872987 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:19.872751    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klz8r\" (UniqueName: \"kubernetes.io/projected/a4854a56-ebca-4747-9e54-a37e5a5eca75-kube-api-access-klz8r\") pod \"payload-processing-e2e-ait-5ff4cfa5-97fcb7787-4dfjj\" (UID: \"a4854a56-ebca-4747-9e54-a37e5a5eca75\") " pod="openshift-ingress/payload-processing-e2e-ait-5ff4cfa5-97fcb7787-4dfjj"
Oct 30 22:40:19.872987 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:19.872787    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4854a56-ebca-4747-9e54-a37e5a5eca75-plugins-config-volume\") pod \"payload-processing-e2e-ait-5ff4cfa5-97fcb7787-4dfjj\" (UID: \"a4854a56-ebca-4747-9e54-a37e5a5eca75\") " pod="openshift-ingress/payload-processing-e2e-ait-5ff4cfa5-97fcb7787-4dfjj"
Oct 30 22:40:19.873530 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:19.873509    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4854a56-ebca-4747-9e54-a37e5a5eca75-plugins-config-volume\") pod \"payload-processing-e2e-ait-5ff4cfa5-97fcb7787-4dfjj\" (UID: \"a4854a56-ebca-4747-9e54-a37e5a5eca75\") " pod="openshift-ingress/payload-processing-e2e-ait-5ff4cfa5-97fcb7787-4dfjj"
Oct 30 22:40:19.930309 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:19.930282    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klz8r\" (UniqueName: \"kubernetes.io/projected/a4854a56-ebca-4747-9e54-a37e5a5eca75-kube-api-access-klz8r\") pod \"payload-processing-e2e-ait-5ff4cfa5-97fcb7787-4dfjj\" (UID: \"a4854a56-ebca-4747-9e54-a37e5a5eca75\") " pod="openshift-ingress/payload-processing-e2e-ait-5ff4cfa5-97fcb7787-4dfjj"
Oct 30 22:40:19.988398 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:19.988368    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5ff4cfa5-97fcb7787-4dfjj"
Oct 30 22:40:20.191603 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:20.191579    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5ff4cfa5-7bdc4f6fb7-ktnhr"]
Oct 30 22:40:20.194313 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:40:20.194285    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0da68f3_69e5_4825_ac6b_f62ade01fe68.slice/crio-7233f05dd5f79d279eeacb988cf37f72253d8415e6ad27243e59fc0e915f3225 WatchSource:0}: Error finding container 7233f05dd5f79d279eeacb988cf37f72253d8415e6ad27243e59fc0e915f3225: Status 404 returned error can't find the container with id 7233f05dd5f79d279eeacb988cf37f72253d8415e6ad27243e59fc0e915f3225
Oct 30 22:40:20.361375 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:20.361332    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5ff4cfa5-97fcb7787-4dfjj"]
Oct 30 22:40:20.363895 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:40:20.363869    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4854a56_ebca_4747_9e54_a37e5a5eca75.slice/crio-41dac86862eaaeb40e59df5c3aae074485bde45004cc93510bcc1e388c8a66cf WatchSource:0}: Error finding container 41dac86862eaaeb40e59df5c3aae074485bde45004cc93510bcc1e388c8a66cf: Status 404 returned error can't find the container with id 41dac86862eaaeb40e59df5c3aae074485bde45004cc93510bcc1e388c8a66cf
Oct 30 22:40:20.451866 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:20.451821    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ff4cfa5-7bdc4f6fb7-ktnhr" event={"ID":"c0da68f3-69e5-4825-ac6b-f62ade01fe68","Type":"ContainerStarted","Data":"4ffd39b153ec716995e351a866dedea5071e02fac862aa7c47a665d62be85fe3"}
Oct 30 22:40:20.452054 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:20.451917    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ff4cfa5-7bdc4f6fb7-ktnhr"
Oct 30 22:40:20.452054 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:20.451946    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ff4cfa5-7bdc4f6fb7-ktnhr" event={"ID":"c0da68f3-69e5-4825-ac6b-f62ade01fe68","Type":"ContainerStarted","Data":"7233f05dd5f79d279eeacb988cf37f72253d8415e6ad27243e59fc0e915f3225"}
Oct 30 22:40:20.454242 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:20.454208    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5ff4cfa5-97fcb7787-4dfjj" event={"ID":"a4854a56-ebca-4747-9e54-a37e5a5eca75","Type":"ContainerStarted","Data":"8a8f639f6ffcecd008319ff21e14908ed126cc1f56f609c9f8e7c1305e9abea3"}
Oct 30 22:40:20.454394 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:20.454264    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5ff4cfa5-97fcb7787-4dfjj" event={"ID":"a4854a56-ebca-4747-9e54-a37e5a5eca75","Type":"ContainerStarted","Data":"41dac86862eaaeb40e59df5c3aae074485bde45004cc93510bcc1e388c8a66cf"}
Oct 30 22:40:20.454394 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:20.454293    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5ff4cfa5-97fcb7787-4dfjj"
Oct 30 22:40:20.569907 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:20.569852    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ff4cfa5-7bdc4f6fb7-ktnhr" podStartSLOduration=1.569836791 podStartE2EDuration="1.569836791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:20.567865542 +0000 UTC m=+4015.042369540" watchObservedRunningTime="2026-10-30 22:40:20.569836791 +0000 UTC m=+4015.044340770"
Oct 30 22:40:20.682340 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:20.682245    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5ff4cfa5-97fcb7787-4dfjj" podStartSLOduration=1.682230386 podStartE2EDuration="1.682230386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:20.682029033 +0000 UTC m=+4015.156533011" watchObservedRunningTime="2026-10-30 22:40:20.682230386 +0000 UTC m=+4015.156734364"
Oct 30 22:40:31.462928 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:31.462900    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ff4cfa5-7bdc4f6fb7-ktnhr"
Oct 30 22:40:31.463363 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:40:31.463077    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5ff4cfa5-97fcb7787-4dfjj"
Oct 30 22:40:47.167511 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:40:47.167472    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:41:02.283892 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:02.283858    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9"]
Oct 30 22:41:02.286989 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:02.286973    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9"
Oct 30 22:41:02.340862 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:02.340838    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:41:02.366410 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:02.366377    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77ce8553-f841-44c9-a0ed-273c3c83d816-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9\" (UID: \"77ce8553-f841-44c9-a0ed-273c3c83d816\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9"
Oct 30 22:41:02.366546 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:02.366426    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c522l\" (UniqueName: \"kubernetes.io/projected/77ce8553-f841-44c9-a0ed-273c3c83d816-kube-api-access-c522l\") pod \"maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9\" (UID: \"77ce8553-f841-44c9-a0ed-273c3c83d816\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9"
Oct 30 22:41:02.366546 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:02.366525    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77ce8553-f841-44c9-a0ed-273c3c83d816-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9\" (UID: \"77ce8553-f841-44c9-a0ed-273c3c83d816\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9"
Oct 30 22:41:02.390290 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:02.390265    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgc9x\""
Oct 30 22:41:02.392177 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:02.392155    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9"]
Oct 30 22:41:02.467835 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:02.467812    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77ce8553-f841-44c9-a0ed-273c3c83d816-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9\" (UID: \"77ce8553-f841-44c9-a0ed-273c3c83d816\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9"
Oct 30 22:41:02.467974 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:02.467851    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c522l\" (UniqueName: \"kubernetes.io/projected/77ce8553-f841-44c9-a0ed-273c3c83d816-kube-api-access-c522l\") pod \"maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9\" (UID: \"77ce8553-f841-44c9-a0ed-273c3c83d816\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9"
Oct 30 22:41:02.467974 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:02.467875    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77ce8553-f841-44c9-a0ed-273c3c83d816-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9\" (UID: \"77ce8553-f841-44c9-a0ed-273c3c83d816\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9"
Oct 30 22:41:02.468544 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:02.468522    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77ce8553-f841-44c9-a0ed-273c3c83d816-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9\" (UID: \"77ce8553-f841-44c9-a0ed-273c3c83d816\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9"
Oct 30 22:41:02.470421 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:02.470403    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77ce8553-f841-44c9-a0ed-273c3c83d816-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9\" (UID: \"77ce8553-f841-44c9-a0ed-273c3c83d816\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9"
Oct 30 22:41:02.525161 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:02.525134    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c522l\" (UniqueName: \"kubernetes.io/projected/77ce8553-f841-44c9-a0ed-273c3c83d816-kube-api-access-c522l\") pod \"maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9\" (UID: \"77ce8553-f841-44c9-a0ed-273c3c83d816\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9"
Oct 30 22:41:02.599159 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:02.599103    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9"
Oct 30 22:41:02.986402 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:02.985814    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9"]
Oct 30 22:41:02.993742 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:41:02.993714    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77ce8553_f841_44c9_a0ed_273c3c83d816.slice/crio-9bf88b71f77f36af068005115633bf4b1f0e7710648baa511374b3852905bbb9 WatchSource:0}: Error finding container 9bf88b71f77f36af068005115633bf4b1f0e7710648baa511374b3852905bbb9: Status 404 returned error can't find the container with id 9bf88b71f77f36af068005115633bf4b1f0e7710648baa511374b3852905bbb9
Oct 30 22:41:03.660017 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:03.659913    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9" event={"ID":"77ce8553-f841-44c9-a0ed-273c3c83d816","Type":"ContainerStarted","Data":"a4d45095e37d3aad5d049824164d89a682b9cab34a1c13f8e6e673669fcecf29"}
Oct 30 22:41:03.660017 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:03.659980    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9" event={"ID":"77ce8553-f841-44c9-a0ed-273c3c83d816","Type":"ContainerStarted","Data":"9bf88b71f77f36af068005115633bf4b1f0e7710648baa511374b3852905bbb9"}
Oct 30 22:41:03.774520 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:03.774474    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9" podStartSLOduration=1.7744615540000002 podStartE2EDuration="1.774461554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:03.772684005 +0000 UTC m=+4058.247187984" watchObservedRunningTime="2026-10-30 22:41:03.774461554 +0000 UTC m=+4058.248965540"
Oct 30 22:41:04.665967 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:04.665934    5087 generic.go:356] "Generic (PLEG): container finished" podID="77ce8553-f841-44c9-a0ed-273c3c83d816" containerID="a4d45095e37d3aad5d049824164d89a682b9cab34a1c13f8e6e673669fcecf29" exitCode=0
Oct 30 22:41:04.666442 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:04.666010    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9" event={"ID":"77ce8553-f841-44c9-a0ed-273c3c83d816","Type":"ContainerDied","Data":"a4d45095e37d3aad5d049824164d89a682b9cab34a1c13f8e6e673669fcecf29"}
Oct 30 22:41:05.809000 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:05.808975    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9"
Oct 30 22:41:05.899116 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:05.899082    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77ce8553-f841-44c9-a0ed-273c3c83d816-kube-api-access-c522l\" (UniqueName: \"kubernetes.io/projected/77ce8553-f841-44c9-a0ed-273c3c83d816-kube-api-access-c522l\") pod \"77ce8553-f841-44c9-a0ed-273c3c83d816\" (UID: \"77ce8553-f841-44c9-a0ed-273c3c83d816\") "
Oct 30 22:41:05.899116 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:05.899119    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77ce8553-f841-44c9-a0ed-273c3c83d816-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77ce8553-f841-44c9-a0ed-273c3c83d816-maas-api-xks-ca\") pod \"77ce8553-f841-44c9-a0ed-273c3c83d816\" (UID: \"77ce8553-f841-44c9-a0ed-273c3c83d816\") "
Oct 30 22:41:05.899424 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:05.899280    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77ce8553-f841-44c9-a0ed-273c3c83d816-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77ce8553-f841-44c9-a0ed-273c3c83d816-maas-api-openshift-ca\") pod \"77ce8553-f841-44c9-a0ed-273c3c83d816\" (UID: \"77ce8553-f841-44c9-a0ed-273c3c83d816\") "
Oct 30 22:41:05.899688 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:05.899661    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77ce8553-f841-44c9-a0ed-273c3c83d816-maas-api-openshift-ca" pod "77ce8553-f841-44c9-a0ed-273c3c83d816" (UID: "77ce8553-f841-44c9-a0ed-273c3c83d816"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:41:05.901462 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:05.901442    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77ce8553-f841-44c9-a0ed-273c3c83d816-maas-api-xks-ca" pod "77ce8553-f841-44c9-a0ed-273c3c83d816" (UID: "77ce8553-f841-44c9-a0ed-273c3c83d816"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:41:05.901533 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:05.901470    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ce8553-f841-44c9-a0ed-273c3c83d816-kube-api-access-c522l" pod "77ce8553-f841-44c9-a0ed-273c3c83d816" (UID: "77ce8553-f841-44c9-a0ed-273c3c83d816"). InnerVolumeSpecName "kube-api-access-c522l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:06.000465 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:06.000428    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77ce8553-f841-44c9-a0ed-273c3c83d816-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:06.000465 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:06.000459    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c522l\" (UniqueName: \"kubernetes.io/projected/77ce8553-f841-44c9-a0ed-273c3c83d816-kube-api-access-c522l\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:06.000465 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:06.000469    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77ce8553-f841-44c9-a0ed-273c3c83d816-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:06.676692 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:06.676658    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9" event={"ID":"77ce8553-f841-44c9-a0ed-273c3c83d816","Type":"ContainerDied","Data":"9bf88b71f77f36af068005115633bf4b1f0e7710648baa511374b3852905bbb9"}
Oct 30 22:41:06.676912 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:06.676695    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9"
Oct 30 22:41:06.676912 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:06.676705    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bf88b71f77f36af068005115633bf4b1f0e7710648baa511374b3852905bbb9"
Oct 30 22:41:12.658199 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:12.658147    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5ff4cfa5-97fcb7787-4dfjj"]
Oct 30 22:41:12.765092 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:12.765056    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5ff4cfa5-7bdc4f6fb7-ktnhr"]
Oct 30 22:41:17.166915 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:41:17.166878    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:41:17.658867 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:17.658824    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5ff4cfa5-97fcb7787-4dfjj" podUID="a4854a56-ebca-4747-9e54-a37e5a5eca75" containerName="payload-processing" containerID="cri-o://8a8f639f6ffcecd008319ff21e14908ed126cc1f56f609c9f8e7c1305e9abea3" gracePeriod=25
Oct 30 22:41:17.766211 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:17.766173    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ff4cfa5-7bdc4f6fb7-ktnhr" podUID="c0da68f3-69e5-4825-ac6b-f62ade01fe68" containerName="payload-pre-processing" containerID="cri-o://4ffd39b153ec716995e351a866dedea5071e02fac862aa7c47a665d62be85fe3" gracePeriod=25
Oct 30 22:41:17.917520 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:17.917495    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5ff4cfa5-97fcb7787-4dfjj"
Oct 30 22:41:18.010788 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:18.010764    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4854a56-ebca-4747-9e54-a37e5a5eca75-kube-api-access-klz8r\" (UniqueName: \"kubernetes.io/projected/a4854a56-ebca-4747-9e54-a37e5a5eca75-kube-api-access-klz8r\") pod \"a4854a56-ebca-4747-9e54-a37e5a5eca75\" (UID: \"a4854a56-ebca-4747-9e54-a37e5a5eca75\") "
Oct 30 22:41:18.010944 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:18.010888    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4854a56-ebca-4747-9e54-a37e5a5eca75-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4854a56-ebca-4747-9e54-a37e5a5eca75-plugins-config-volume\") pod \"a4854a56-ebca-4747-9e54-a37e5a5eca75\" (UID: \"a4854a56-ebca-4747-9e54-a37e5a5eca75\") "
Oct 30 22:41:18.011546 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:18.011522    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4854a56-ebca-4747-9e54-a37e5a5eca75-plugins-config-volume" pod "a4854a56-ebca-4747-9e54-a37e5a5eca75" (UID: "a4854a56-ebca-4747-9e54-a37e5a5eca75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:41:18.013649 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:18.013626    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4854a56-ebca-4747-9e54-a37e5a5eca75-kube-api-access-klz8r" pod "a4854a56-ebca-4747-9e54-a37e5a5eca75" (UID: "a4854a56-ebca-4747-9e54-a37e5a5eca75"). InnerVolumeSpecName "kube-api-access-klz8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:18.017565 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:18.017519    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ff4cfa5-7bdc4f6fb7-ktnhr"
Oct 30 22:41:18.111962 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:18.111930    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0da68f3-69e5-4825-ac6b-f62ade01fe68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0da68f3-69e5-4825-ac6b-f62ade01fe68-plugins-config-volume\") pod \"c0da68f3-69e5-4825-ac6b-f62ade01fe68\" (UID: \"c0da68f3-69e5-4825-ac6b-f62ade01fe68\") "
Oct 30 22:41:18.112136 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:18.112066    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0da68f3-69e5-4825-ac6b-f62ade01fe68-kube-api-access-bvv8k\" (UniqueName: \"kubernetes.io/projected/c0da68f3-69e5-4825-ac6b-f62ade01fe68-kube-api-access-bvv8k\") pod \"c0da68f3-69e5-4825-ac6b-f62ade01fe68\" (UID: \"c0da68f3-69e5-4825-ac6b-f62ade01fe68\") "
Oct 30 22:41:18.112322 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:18.112305    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klz8r\" (UniqueName: \"kubernetes.io/projected/a4854a56-ebca-4747-9e54-a37e5a5eca75-kube-api-access-klz8r\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:18.112418 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:18.112326    5087 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4854a56-ebca-4747-9e54-a37e5a5eca75-plugins-config-volume\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:18.112418 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:18.112393    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0da68f3-69e5-4825-ac6b-f62ade01fe68-plugins-config-volume" pod "c0da68f3-69e5-4825-ac6b-f62ade01fe68" (UID: "c0da68f3-69e5-4825-ac6b-f62ade01fe68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:41:18.114210 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:18.114190    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0da68f3-69e5-4825-ac6b-f62ade01fe68-kube-api-access-bvv8k" pod "c0da68f3-69e5-4825-ac6b-f62ade01fe68" (UID: "c0da68f3-69e5-4825-ac6b-f62ade01fe68"). InnerVolumeSpecName "kube-api-access-bvv8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:18.213205 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:18.213173    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvv8k\" (UniqueName: \"kubernetes.io/projected/c0da68f3-69e5-4825-ac6b-f62ade01fe68-kube-api-access-bvv8k\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:18.213205 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:18.213197    5087 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0da68f3-69e5-4825-ac6b-f62ade01fe68-plugins-config-volume\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:41:18.734503 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:18.734469    5087 generic.go:356] "Generic (PLEG): container finished" podID="c0da68f3-69e5-4825-ac6b-f62ade01fe68" containerID="4ffd39b153ec716995e351a866dedea5071e02fac862aa7c47a665d62be85fe3" exitCode=0
Oct 30 22:41:18.734692 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:18.734522    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ff4cfa5-7bdc4f6fb7-ktnhr"
Oct 30 22:41:18.734692 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:18.734552    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ff4cfa5-7bdc4f6fb7-ktnhr" event={"ID":"c0da68f3-69e5-4825-ac6b-f62ade01fe68","Type":"ContainerDied","Data":"4ffd39b153ec716995e351a866dedea5071e02fac862aa7c47a665d62be85fe3"}
Oct 30 22:41:18.734692 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:18.734612    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5ff4cfa5-7bdc4f6fb7-ktnhr" event={"ID":"c0da68f3-69e5-4825-ac6b-f62ade01fe68","Type":"ContainerDied","Data":"7233f05dd5f79d279eeacb988cf37f72253d8415e6ad27243e59fc0e915f3225"}
Oct 30 22:41:18.734692 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:18.734644    5087 scope.go:122] "RemoveContainer" containerID="4ffd39b153ec716995e351a866dedea5071e02fac862aa7c47a665d62be85fe3"
Oct 30 22:41:18.736021 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:18.735947    5087 generic.go:356] "Generic (PLEG): container finished" podID="a4854a56-ebca-4747-9e54-a37e5a5eca75" containerID="8a8f639f6ffcecd008319ff21e14908ed126cc1f56f609c9f8e7c1305e9abea3" exitCode=0
Oct 30 22:41:18.736021 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:18.735988    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5ff4cfa5-97fcb7787-4dfjj"
Oct 30 22:41:18.736135 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:18.736021    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5ff4cfa5-97fcb7787-4dfjj" event={"ID":"a4854a56-ebca-4747-9e54-a37e5a5eca75","Type":"ContainerDied","Data":"8a8f639f6ffcecd008319ff21e14908ed126cc1f56f609c9f8e7c1305e9abea3"}
Oct 30 22:41:18.736135 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:18.736070    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5ff4cfa5-97fcb7787-4dfjj" event={"ID":"a4854a56-ebca-4747-9e54-a37e5a5eca75","Type":"ContainerDied","Data":"41dac86862eaaeb40e59df5c3aae074485bde45004cc93510bcc1e388c8a66cf"}
Oct 30 22:41:18.744650 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:18.744630    5087 scope.go:122] "RemoveContainer" containerID="4ffd39b153ec716995e351a866dedea5071e02fac862aa7c47a665d62be85fe3"
Oct 30 22:41:18.745023 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:41:18.745000    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ffd39b153ec716995e351a866dedea5071e02fac862aa7c47a665d62be85fe3\": container with ID starting with 4ffd39b153ec716995e351a866dedea5071e02fac862aa7c47a665d62be85fe3 not found: ID does not exist" containerID="4ffd39b153ec716995e351a866dedea5071e02fac862aa7c47a665d62be85fe3"
Oct 30 22:41:18.745127 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:18.745030    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ffd39b153ec716995e351a866dedea5071e02fac862aa7c47a665d62be85fe3"} err="failed to get container status \"4ffd39b153ec716995e351a866dedea5071e02fac862aa7c47a665d62be85fe3\": rpc error: code = NotFound desc = could not find container \"4ffd39b153ec716995e351a866dedea5071e02fac862aa7c47a665d62be85fe3\": container with ID starting with 4ffd39b153ec716995e351a866dedea5071e02fac862aa7c47a665d62be85fe3 not found: ID does not exist"
Oct 30 22:41:18.745127 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:18.745049    5087 scope.go:122] "RemoveContainer" containerID="8a8f639f6ffcecd008319ff21e14908ed126cc1f56f609c9f8e7c1305e9abea3"
Oct 30 22:41:18.753075 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:18.753060    5087 scope.go:122] "RemoveContainer" containerID="8a8f639f6ffcecd008319ff21e14908ed126cc1f56f609c9f8e7c1305e9abea3"
Oct 30 22:41:18.753333 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:41:18.753314    5087 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a8f639f6ffcecd008319ff21e14908ed126cc1f56f609c9f8e7c1305e9abea3\": container with ID starting with 8a8f639f6ffcecd008319ff21e14908ed126cc1f56f609c9f8e7c1305e9abea3 not found: ID does not exist" containerID="8a8f639f6ffcecd008319ff21e14908ed126cc1f56f609c9f8e7c1305e9abea3"
Oct 30 22:41:18.753401 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:18.753341    5087 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a8f639f6ffcecd008319ff21e14908ed126cc1f56f609c9f8e7c1305e9abea3"} err="failed to get container status \"8a8f639f6ffcecd008319ff21e14908ed126cc1f56f609c9f8e7c1305e9abea3\": rpc error: code = NotFound desc = could not find container \"8a8f639f6ffcecd008319ff21e14908ed126cc1f56f609c9f8e7c1305e9abea3\": container with ID starting with 8a8f639f6ffcecd008319ff21e14908ed126cc1f56f609c9f8e7c1305e9abea3 not found: ID does not exist"
Oct 30 22:41:18.902941 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:18.902728    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5ff4cfa5-97fcb7787-4dfjj"]
Oct 30 22:41:18.908616 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:18.908590    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5ff4cfa5-97fcb7787-4dfjj"]
Oct 30 22:41:19.060651 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:19.060613    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5ff4cfa5-7bdc4f6fb7-ktnhr"]
Oct 30 22:41:19.064741 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:19.064715    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5ff4cfa5-7bdc4f6fb7-ktnhr"]
Oct 30 22:41:19.708390 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:19.708341    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4854a56-ebca-4747-9e54-a37e5a5eca75" path="/var/lib/kubelet/pods/a4854a56-ebca-4747-9e54-a37e5a5eca75/volumes"
Oct 30 22:41:19.708903 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:19.708854    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0da68f3-69e5-4825-ac6b-f62ade01fe68" path="/var/lib/kubelet/pods/c0da68f3-69e5-4825-ac6b-f62ade01fe68/volumes"
Oct 30 22:41:20.926740 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:20.926705    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9"]
Oct 30 22:41:20.977606 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:20.977570    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5ff4cfa5-7ztm9"]
Oct 30 22:41:21.707687 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:41:21.707655    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77ce8553-f841-44c9-a0ed-273c3c83d816" path="/var/lib/kubelet/pods/77ce8553-f841-44c9-a0ed-273c3c83d816/volumes"
Oct 30 22:41:47.167175 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:41:47.167098    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:42:17.167198 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:42:17.167159    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:42:47.166716 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:42:47.166679    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:43:17.166996 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:43:17.166909    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:43:25.194918 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:25.194881    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v"]
Oct 30 22:43:25.195434 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:25.195279    5087 cpu_manager.go:414] "containerMap: removing container" podUID="c0da68f3-69e5-4825-ac6b-f62ade01fe68" containerName="payload-pre-processing"
Oct 30 22:43:25.195434 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:25.195291    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0da68f3-69e5-4825-ac6b-f62ade01fe68" containerName="payload-pre-processing"
Oct 30 22:43:25.195434 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:25.195314    5087 cpu_manager.go:414] "containerMap: removing container" podUID="a4854a56-ebca-4747-9e54-a37e5a5eca75" containerName="payload-processing"
Oct 30 22:43:25.195434 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:25.195319    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4854a56-ebca-4747-9e54-a37e5a5eca75" containerName="payload-processing"
Oct 30 22:43:25.195434 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:25.195326    5087 cpu_manager.go:414] "containerMap: removing container" podUID="77ce8553-f841-44c9-a0ed-273c3c83d816" containerName="revoke-keys"
Oct 30 22:43:25.195434 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:25.195331    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77ce8553-f841-44c9-a0ed-273c3c83d816" containerName="revoke-keys"
Oct 30 22:43:25.195434 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:25.195414    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4854a56-ebca-4747-9e54-a37e5a5eca75" containerName="payload-processing"
Oct 30 22:43:25.195434 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:25.195424    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="77ce8553-f841-44c9-a0ed-273c3c83d816" containerName="revoke-keys"
Oct 30 22:43:25.195434 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:25.195435    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0da68f3-69e5-4825-ac6b-f62ade01fe68" containerName="payload-pre-processing"
Oct 30 22:43:25.198643 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:25.198628    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v"
Oct 30 22:43:25.256078 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:25.256037    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v"]
Oct 30 22:43:25.290573 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:25.290541    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ab10217-027c-4fc6-b431-037e9ee4866f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v\" (UID: \"6ab10217-027c-4fc6-b431-037e9ee4866f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v"
Oct 30 22:43:25.290734 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:25.290594    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s24wl\" (UniqueName: \"kubernetes.io/projected/6ab10217-027c-4fc6-b431-037e9ee4866f-kube-api-access-s24wl\") pod \"maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v\" (UID: \"6ab10217-027c-4fc6-b431-037e9ee4866f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v"
Oct 30 22:43:25.290734 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:25.290721    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ab10217-027c-4fc6-b431-037e9ee4866f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v\" (UID: \"6ab10217-027c-4fc6-b431-037e9ee4866f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v"
Oct 30 22:43:25.302528 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:25.302504    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:43:25.302669 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:25.302596    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgc9x\""
Oct 30 22:43:25.391512 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:25.391481    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ab10217-027c-4fc6-b431-037e9ee4866f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v\" (UID: \"6ab10217-027c-4fc6-b431-037e9ee4866f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v"
Oct 30 22:43:25.391687 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:25.391522    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s24wl\" (UniqueName: \"kubernetes.io/projected/6ab10217-027c-4fc6-b431-037e9ee4866f-kube-api-access-s24wl\") pod \"maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v\" (UID: \"6ab10217-027c-4fc6-b431-037e9ee4866f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v"
Oct 30 22:43:25.391687 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:25.391578    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ab10217-027c-4fc6-b431-037e9ee4866f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v\" (UID: \"6ab10217-027c-4fc6-b431-037e9ee4866f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v"
Oct 30 22:43:25.392338 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:25.392311    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ab10217-027c-4fc6-b431-037e9ee4866f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v\" (UID: \"6ab10217-027c-4fc6-b431-037e9ee4866f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v"
Oct 30 22:43:25.394101 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:25.394078    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ab10217-027c-4fc6-b431-037e9ee4866f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v\" (UID: \"6ab10217-027c-4fc6-b431-037e9ee4866f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v"
Oct 30 22:43:25.452463 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:25.452406    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s24wl\" (UniqueName: \"kubernetes.io/projected/6ab10217-027c-4fc6-b431-037e9ee4866f-kube-api-access-s24wl\") pod \"maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v\" (UID: \"6ab10217-027c-4fc6-b431-037e9ee4866f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v"
Oct 30 22:43:25.511669 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:25.511647    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v"
Oct 30 22:43:25.889059 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:25.889038    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v"]
Oct 30 22:43:25.891517 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:43:25.891486    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab10217_027c_4fc6_b431_037e9ee4866f.slice/crio-80349e04d2a5f3982a3005212451808c3af2f5eb8f7b6c61bd5e7c67a71b660e WatchSource:0}: Error finding container 80349e04d2a5f3982a3005212451808c3af2f5eb8f7b6c61bd5e7c67a71b660e: Status 404 returned error can't find the container with id 80349e04d2a5f3982a3005212451808c3af2f5eb8f7b6c61bd5e7c67a71b660e
Oct 30 22:43:26.345668 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:26.345628    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v" event={"ID":"6ab10217-027c-4fc6-b431-037e9ee4866f","Type":"ContainerStarted","Data":"bdb15dfbb86436bdc6cca8253a60b96932431fa755e95bedb4cbe6e53fa89e29"}
Oct 30 22:43:26.346118 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:26.345722    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v" event={"ID":"6ab10217-027c-4fc6-b431-037e9ee4866f","Type":"ContainerStarted","Data":"80349e04d2a5f3982a3005212451808c3af2f5eb8f7b6c61bd5e7c67a71b660e"}
Oct 30 22:43:26.353318 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:26.353293    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 22:43:26.357119 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:26.357100    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 22:43:26.361771 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:26.361751    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 22:43:26.365703 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:26.365687    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 22:43:26.466530 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:26.466433    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v" podStartSLOduration=1.466419647 podStartE2EDuration="1.466419647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:26.464384398 +0000 UTC m=+4200.938888377" watchObservedRunningTime="2026-10-30 22:43:26.466419647 +0000 UTC m=+4200.940923626"
Oct 30 22:43:27.352431 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:27.352397    5087 generic.go:356] "Generic (PLEG): container finished" podID="6ab10217-027c-4fc6-b431-037e9ee4866f" containerID="bdb15dfbb86436bdc6cca8253a60b96932431fa755e95bedb4cbe6e53fa89e29" exitCode=0
Oct 30 22:43:27.352843 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:27.352486    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v" event={"ID":"6ab10217-027c-4fc6-b431-037e9ee4866f","Type":"ContainerDied","Data":"bdb15dfbb86436bdc6cca8253a60b96932431fa755e95bedb4cbe6e53fa89e29"}
Oct 30 22:43:28.499480 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:28.499457    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v"
Oct 30 22:43:28.621282 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:28.621196    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ab10217-027c-4fc6-b431-037e9ee4866f-kube-api-access-s24wl\" (UniqueName: \"kubernetes.io/projected/6ab10217-027c-4fc6-b431-037e9ee4866f-kube-api-access-s24wl\") pod \"6ab10217-027c-4fc6-b431-037e9ee4866f\" (UID: \"6ab10217-027c-4fc6-b431-037e9ee4866f\") "
Oct 30 22:43:28.621282 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:28.621246    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ab10217-027c-4fc6-b431-037e9ee4866f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ab10217-027c-4fc6-b431-037e9ee4866f-maas-api-xks-ca\") pod \"6ab10217-027c-4fc6-b431-037e9ee4866f\" (UID: \"6ab10217-027c-4fc6-b431-037e9ee4866f\") "
Oct 30 22:43:28.621282 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:28.621276    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ab10217-027c-4fc6-b431-037e9ee4866f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ab10217-027c-4fc6-b431-037e9ee4866f-maas-api-openshift-ca\") pod \"6ab10217-027c-4fc6-b431-037e9ee4866f\" (UID: \"6ab10217-027c-4fc6-b431-037e9ee4866f\") "
Oct 30 22:43:28.621759 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:28.621734    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ab10217-027c-4fc6-b431-037e9ee4866f-maas-api-openshift-ca" pod "6ab10217-027c-4fc6-b431-037e9ee4866f" (UID: "6ab10217-027c-4fc6-b431-037e9ee4866f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:43:28.624169 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:28.624136    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ab10217-027c-4fc6-b431-037e9ee4866f-maas-api-xks-ca" pod "6ab10217-027c-4fc6-b431-037e9ee4866f" (UID: "6ab10217-027c-4fc6-b431-037e9ee4866f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:43:28.624324 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:28.624167    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab10217-027c-4fc6-b431-037e9ee4866f-kube-api-access-s24wl" pod "6ab10217-027c-4fc6-b431-037e9ee4866f" (UID: "6ab10217-027c-4fc6-b431-037e9ee4866f"). InnerVolumeSpecName "kube-api-access-s24wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:28.722211 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:28.722175    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ab10217-027c-4fc6-b431-037e9ee4866f-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:43:28.722211 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:28.722203    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ab10217-027c-4fc6-b431-037e9ee4866f-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:43:28.722211 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:28.722214    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s24wl\" (UniqueName: \"kubernetes.io/projected/6ab10217-027c-4fc6-b431-037e9ee4866f-kube-api-access-s24wl\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:43:29.363413 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:29.363377    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v"
Oct 30 22:43:29.363594 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:29.363410    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v" event={"ID":"6ab10217-027c-4fc6-b431-037e9ee4866f","Type":"ContainerDied","Data":"80349e04d2a5f3982a3005212451808c3af2f5eb8f7b6c61bd5e7c67a71b660e"}
Oct 30 22:43:29.363594 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:29.363460    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80349e04d2a5f3982a3005212451808c3af2f5eb8f7b6c61bd5e7c67a71b660e"
Oct 30 22:43:47.167856 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:43:47.167818    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:43:49.902126 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:49.902091    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v"]
Oct 30 22:43:49.951340 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:49.951304    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-f0b1ab-v726v"]
Oct 30 22:43:51.712109 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:43:51.712070    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ab10217-027c-4fc6-b431-037e9ee4866f" path="/var/lib/kubelet/pods/6ab10217-027c-4fc6-b431-037e9ee4866f/volumes"
Oct 30 22:44:17.172391 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:44:17.172334    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:44:24.139455 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:24.139420    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-266w9"]
Oct 30 22:44:24.139867 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:24.139829    5087 cpu_manager.go:414] "containerMap: removing container" podUID="6ab10217-027c-4fc6-b431-037e9ee4866f" containerName="revoke-keys"
Oct 30 22:44:24.139867 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:24.139841    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ab10217-027c-4fc6-b431-037e9ee4866f" containerName="revoke-keys"
Oct 30 22:44:24.139946 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:24.139908    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ab10217-027c-4fc6-b431-037e9ee4866f" containerName="revoke-keys"
Oct 30 22:44:24.142963 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:24.142947    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-266w9"
Oct 30 22:44:24.169474 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:24.169444    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dft62\" (UniqueName: \"kubernetes.io/projected/07407208-6eca-43c8-936b-8cbcb88283cf-kube-api-access-dft62\") pod \"maas-api-revoke-sub-f03629db-266w9\" (UID: \"07407208-6eca-43c8-936b-8cbcb88283cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-266w9"
Oct 30 22:44:24.169613 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:24.169482    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07407208-6eca-43c8-936b-8cbcb88283cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-266w9\" (UID: \"07407208-6eca-43c8-936b-8cbcb88283cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-266w9"
Oct 30 22:44:24.169657 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:24.169632    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07407208-6eca-43c8-936b-8cbcb88283cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-266w9\" (UID: \"07407208-6eca-43c8-936b-8cbcb88283cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-266w9"
Oct 30 22:44:24.197141 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:24.197107    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:44:24.246608 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:24.246577    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgc9x\""
Oct 30 22:44:24.248393 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:24.248374    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-266w9"]
Oct 30 22:44:24.270583 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:24.270555    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dft62\" (UniqueName: \"kubernetes.io/projected/07407208-6eca-43c8-936b-8cbcb88283cf-kube-api-access-dft62\") pod \"maas-api-revoke-sub-f03629db-266w9\" (UID: \"07407208-6eca-43c8-936b-8cbcb88283cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-266w9"
Oct 30 22:44:24.270727 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:24.270594    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07407208-6eca-43c8-936b-8cbcb88283cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-266w9\" (UID: \"07407208-6eca-43c8-936b-8cbcb88283cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-266w9"
Oct 30 22:44:24.270727 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:24.270668    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07407208-6eca-43c8-936b-8cbcb88283cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-266w9\" (UID: \"07407208-6eca-43c8-936b-8cbcb88283cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-266w9"
Oct 30 22:44:24.271268 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:24.271243    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07407208-6eca-43c8-936b-8cbcb88283cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-266w9\" (UID: \"07407208-6eca-43c8-936b-8cbcb88283cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-266w9"
Oct 30 22:44:24.273133 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:24.273112    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07407208-6eca-43c8-936b-8cbcb88283cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-266w9\" (UID: \"07407208-6eca-43c8-936b-8cbcb88283cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-266w9"
Oct 30 22:44:24.326754 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:24.326726    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dft62\" (UniqueName: \"kubernetes.io/projected/07407208-6eca-43c8-936b-8cbcb88283cf-kube-api-access-dft62\") pod \"maas-api-revoke-sub-f03629db-266w9\" (UID: \"07407208-6eca-43c8-936b-8cbcb88283cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-266w9"
Oct 30 22:44:24.455433 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:24.455361    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-266w9"
Oct 30 22:44:24.828046 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:24.828017    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-266w9"]
Oct 30 22:44:24.830275 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:44:24.830250    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07407208_6eca_43c8_936b_8cbcb88283cf.slice/crio-869881b64bf9533d950a699de75ce5393b7ffab1399ef1882458c6981196601e WatchSource:0}: Error finding container 869881b64bf9533d950a699de75ce5393b7ffab1399ef1882458c6981196601e: Status 404 returned error can't find the container with id 869881b64bf9533d950a699de75ce5393b7ffab1399ef1882458c6981196601e
Oct 30 22:44:25.655680 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:25.655646    5087 generic.go:356] "Generic (PLEG): container finished" podID="07407208-6eca-43c8-936b-8cbcb88283cf" containerID="68d5d30aebb5ea3082e17506a512bacbf22bef8e02b4196c9d89ce5a0b893047" exitCode=0
Oct 30 22:44:25.656078 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:25.655721    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-266w9" event={"ID":"07407208-6eca-43c8-936b-8cbcb88283cf","Type":"ContainerDied","Data":"68d5d30aebb5ea3082e17506a512bacbf22bef8e02b4196c9d89ce5a0b893047"}
Oct 30 22:44:25.656078 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:25.655760    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-266w9" event={"ID":"07407208-6eca-43c8-936b-8cbcb88283cf","Type":"ContainerStarted","Data":"869881b64bf9533d950a699de75ce5393b7ffab1399ef1882458c6981196601e"}
Oct 30 22:44:26.805069 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:26.805040    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-266w9"
Oct 30 22:44:26.894068 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:26.894039    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07407208-6eca-43c8-936b-8cbcb88283cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07407208-6eca-43c8-936b-8cbcb88283cf-maas-api-openshift-ca\") pod \"07407208-6eca-43c8-936b-8cbcb88283cf\" (UID: \"07407208-6eca-43c8-936b-8cbcb88283cf\") "
Oct 30 22:44:26.894241 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:26.894092    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07407208-6eca-43c8-936b-8cbcb88283cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07407208-6eca-43c8-936b-8cbcb88283cf-maas-api-xks-ca\") pod \"07407208-6eca-43c8-936b-8cbcb88283cf\" (UID: \"07407208-6eca-43c8-936b-8cbcb88283cf\") "
Oct 30 22:44:26.894241 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:26.894159    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07407208-6eca-43c8-936b-8cbcb88283cf-kube-api-access-dft62\" (UniqueName: \"kubernetes.io/projected/07407208-6eca-43c8-936b-8cbcb88283cf-kube-api-access-dft62\") pod \"07407208-6eca-43c8-936b-8cbcb88283cf\" (UID: \"07407208-6eca-43c8-936b-8cbcb88283cf\") "
Oct 30 22:44:26.894409 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:26.894387    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07407208-6eca-43c8-936b-8cbcb88283cf-maas-api-openshift-ca" pod "07407208-6eca-43c8-936b-8cbcb88283cf" (UID: "07407208-6eca-43c8-936b-8cbcb88283cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:44:26.894510 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:26.894493    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07407208-6eca-43c8-936b-8cbcb88283cf-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:26.896246 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:26.896227    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07407208-6eca-43c8-936b-8cbcb88283cf-maas-api-xks-ca" pod "07407208-6eca-43c8-936b-8cbcb88283cf" (UID: "07407208-6eca-43c8-936b-8cbcb88283cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:44:26.896373 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:26.896307    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07407208-6eca-43c8-936b-8cbcb88283cf-kube-api-access-dft62" pod "07407208-6eca-43c8-936b-8cbcb88283cf" (UID: "07407208-6eca-43c8-936b-8cbcb88283cf"). InnerVolumeSpecName "kube-api-access-dft62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:26.995823 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:26.995800    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07407208-6eca-43c8-936b-8cbcb88283cf-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:26.995823 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:26.995825    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dft62\" (UniqueName: \"kubernetes.io/projected/07407208-6eca-43c8-936b-8cbcb88283cf-kube-api-access-dft62\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:27.667942 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:27.667912    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-266w9"
Oct 30 22:44:27.668146 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:27.667918    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-266w9" event={"ID":"07407208-6eca-43c8-936b-8cbcb88283cf","Type":"ContainerDied","Data":"869881b64bf9533d950a699de75ce5393b7ffab1399ef1882458c6981196601e"}
Oct 30 22:44:27.668146 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:27.668037    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="869881b64bf9533d950a699de75ce5393b7ffab1399ef1882458c6981196601e"
Oct 30 22:44:44.868371 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:44.868317    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4dg8"]
Oct 30 22:44:44.868790 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:44.868732    5087 cpu_manager.go:414] "containerMap: removing container" podUID="07407208-6eca-43c8-936b-8cbcb88283cf" containerName="revoke-keys"
Oct 30 22:44:44.868790 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:44.868745    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07407208-6eca-43c8-936b-8cbcb88283cf" containerName="revoke-keys"
Oct 30 22:44:44.868878 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:44.868828    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="07407208-6eca-43c8-936b-8cbcb88283cf" containerName="revoke-keys"
Oct 30 22:44:44.872262 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:44.872241    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4dg8"
Oct 30 22:44:44.928040 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:44.928009    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4dg8"]
Oct 30 22:44:44.974989 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:44.974957    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:44:44.974989 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:44.974975    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgc9x\""
Oct 30 22:44:44.994472 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:44.994438    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26516f1e-739c-427b-98cd-4e387fd462bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-v4dg8\" (UID: \"26516f1e-739c-427b-98cd-4e387fd462bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4dg8"
Oct 30 22:44:44.994637 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:44.994486    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26516f1e-739c-427b-98cd-4e387fd462bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-v4dg8\" (UID: \"26516f1e-739c-427b-98cd-4e387fd462bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4dg8"
Oct 30 22:44:44.994637 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:44.994515    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7b4l\" (UniqueName: \"kubernetes.io/projected/26516f1e-739c-427b-98cd-4e387fd462bb-kube-api-access-f7b4l\") pod \"maas-api-revoke-sub-0165d7d2-v4dg8\" (UID: \"26516f1e-739c-427b-98cd-4e387fd462bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4dg8"
Oct 30 22:44:45.095684 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:45.095653    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26516f1e-739c-427b-98cd-4e387fd462bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-v4dg8\" (UID: \"26516f1e-739c-427b-98cd-4e387fd462bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4dg8"
Oct 30 22:44:45.095849 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:45.095697    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26516f1e-739c-427b-98cd-4e387fd462bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-v4dg8\" (UID: \"26516f1e-739c-427b-98cd-4e387fd462bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4dg8"
Oct 30 22:44:45.095849 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:45.095716    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7b4l\" (UniqueName: \"kubernetes.io/projected/26516f1e-739c-427b-98cd-4e387fd462bb-kube-api-access-f7b4l\") pod \"maas-api-revoke-sub-0165d7d2-v4dg8\" (UID: \"26516f1e-739c-427b-98cd-4e387fd462bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4dg8"
Oct 30 22:44:45.096455 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:45.096435    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26516f1e-739c-427b-98cd-4e387fd462bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-v4dg8\" (UID: \"26516f1e-739c-427b-98cd-4e387fd462bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4dg8"
Oct 30 22:44:45.098246 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:45.098226    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26516f1e-739c-427b-98cd-4e387fd462bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-v4dg8\" (UID: \"26516f1e-739c-427b-98cd-4e387fd462bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4dg8"
Oct 30 22:44:45.152129 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:45.152057    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7b4l\" (UniqueName: \"kubernetes.io/projected/26516f1e-739c-427b-98cd-4e387fd462bb-kube-api-access-f7b4l\") pod \"maas-api-revoke-sub-0165d7d2-v4dg8\" (UID: \"26516f1e-739c-427b-98cd-4e387fd462bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4dg8"
Oct 30 22:44:45.185748 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:45.185723    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4dg8"
Oct 30 22:44:45.553131 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:45.553107    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4dg8"]
Oct 30 22:44:45.555912 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:44:45.555887    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26516f1e_739c_427b_98cd_4e387fd462bb.slice/crio-0619eb1e5390550fb54d6965837061a8513cd368f80c2f994d873b15bc3a3579 WatchSource:0}: Error finding container 0619eb1e5390550fb54d6965837061a8513cd368f80c2f994d873b15bc3a3579: Status 404 returned error can't find the container with id 0619eb1e5390550fb54d6965837061a8513cd368f80c2f994d873b15bc3a3579
Oct 30 22:44:45.774642 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:45.774611    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4dg8" event={"ID":"26516f1e-739c-427b-98cd-4e387fd462bb","Type":"ContainerStarted","Data":"0619eb1e5390550fb54d6965837061a8513cd368f80c2f994d873b15bc3a3579"}
Oct 30 22:44:46.780874 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:46.780833    5087 generic.go:356] "Generic (PLEG): container finished" podID="26516f1e-739c-427b-98cd-4e387fd462bb" containerID="2a1fdff2d2500b3ca591b0f967972c3a33d25f8db648fb4403f9eacef54d4613" exitCode=0
Oct 30 22:44:46.781280 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:46.780926    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4dg8" event={"ID":"26516f1e-739c-427b-98cd-4e387fd462bb","Type":"ContainerDied","Data":"2a1fdff2d2500b3ca591b0f967972c3a33d25f8db648fb4403f9eacef54d4613"}
Oct 30 22:44:47.167464 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:44:47.167376    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:44:47.924160 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:47.924134    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4dg8"
Oct 30 22:44:48.023750 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:48.023724    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26516f1e-739c-427b-98cd-4e387fd462bb-kube-api-access-f7b4l\" (UniqueName: \"kubernetes.io/projected/26516f1e-739c-427b-98cd-4e387fd462bb-kube-api-access-f7b4l\") pod \"26516f1e-739c-427b-98cd-4e387fd462bb\" (UID: \"26516f1e-739c-427b-98cd-4e387fd462bb\") "
Oct 30 22:44:48.023988 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:48.023776    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26516f1e-739c-427b-98cd-4e387fd462bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26516f1e-739c-427b-98cd-4e387fd462bb-maas-api-xks-ca\") pod \"26516f1e-739c-427b-98cd-4e387fd462bb\" (UID: \"26516f1e-739c-427b-98cd-4e387fd462bb\") "
Oct 30 22:44:48.023988 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:48.023861    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26516f1e-739c-427b-98cd-4e387fd462bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26516f1e-739c-427b-98cd-4e387fd462bb-maas-api-openshift-ca\") pod \"26516f1e-739c-427b-98cd-4e387fd462bb\" (UID: \"26516f1e-739c-427b-98cd-4e387fd462bb\") "
Oct 30 22:44:48.024231 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:48.024204    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26516f1e-739c-427b-98cd-4e387fd462bb-maas-api-openshift-ca" pod "26516f1e-739c-427b-98cd-4e387fd462bb" (UID: "26516f1e-739c-427b-98cd-4e387fd462bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:44:48.025983 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:48.025952    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26516f1e-739c-427b-98cd-4e387fd462bb-maas-api-xks-ca" pod "26516f1e-739c-427b-98cd-4e387fd462bb" (UID: "26516f1e-739c-427b-98cd-4e387fd462bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:44:48.026085 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:48.025984    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26516f1e-739c-427b-98cd-4e387fd462bb-kube-api-access-f7b4l" pod "26516f1e-739c-427b-98cd-4e387fd462bb" (UID: "26516f1e-739c-427b-98cd-4e387fd462bb"). InnerVolumeSpecName "kube-api-access-f7b4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:44:48.125594 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:48.125510    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26516f1e-739c-427b-98cd-4e387fd462bb-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:48.125594 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:48.125538    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7b4l\" (UniqueName: \"kubernetes.io/projected/26516f1e-739c-427b-98cd-4e387fd462bb-kube-api-access-f7b4l\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:48.125594 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:48.125554    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26516f1e-739c-427b-98cd-4e387fd462bb-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:44:48.792005 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:48.791971    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4dg8"
Oct 30 22:44:48.792193 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:48.792004    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4dg8" event={"ID":"26516f1e-739c-427b-98cd-4e387fd462bb","Type":"ContainerDied","Data":"0619eb1e5390550fb54d6965837061a8513cd368f80c2f994d873b15bc3a3579"}
Oct 30 22:44:48.792193 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:48.792051    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0619eb1e5390550fb54d6965837061a8513cd368f80c2f994d873b15bc3a3579"
Oct 30 22:44:56.588712 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:56.588681    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hp9zg"]
Oct 30 22:44:56.589113 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:56.589092    5087 cpu_manager.go:414] "containerMap: removing container" podUID="26516f1e-739c-427b-98cd-4e387fd462bb" containerName="revoke-keys"
Oct 30 22:44:56.589113 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:56.589103    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26516f1e-739c-427b-98cd-4e387fd462bb" containerName="revoke-keys"
Oct 30 22:44:56.589198 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:56.589175    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="26516f1e-739c-427b-98cd-4e387fd462bb" containerName="revoke-keys"
Oct 30 22:44:56.592427 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:56.592409    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hp9zg"
Oct 30 22:44:56.604926 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:56.604889    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbeb3764-7ae9-41d9-8a2a-b6f14affeb06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-hp9zg\" (UID: \"fbeb3764-7ae9-41d9-8a2a-b6f14affeb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hp9zg"
Oct 30 22:44:56.605057 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:56.604929    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kjm8\" (UniqueName: \"kubernetes.io/projected/fbeb3764-7ae9-41d9-8a2a-b6f14affeb06-kube-api-access-6kjm8\") pod \"maas-api-revoke-sub-6b19338f-hp9zg\" (UID: \"fbeb3764-7ae9-41d9-8a2a-b6f14affeb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hp9zg"
Oct 30 22:44:56.605057 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:56.605033    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbeb3764-7ae9-41d9-8a2a-b6f14affeb06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-hp9zg\" (UID: \"fbeb3764-7ae9-41d9-8a2a-b6f14affeb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hp9zg"
Oct 30 22:44:56.644773 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:56.644745    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:44:56.693976 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:56.693948    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-wgc9x\""
Oct 30 22:44:56.695937 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:56.695915    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hp9zg"]
Oct 30 22:44:56.705796 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:56.705560    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbeb3764-7ae9-41d9-8a2a-b6f14affeb06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-hp9zg\" (UID: \"fbeb3764-7ae9-41d9-8a2a-b6f14affeb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hp9zg"
Oct 30 22:44:56.705796 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:56.705596    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kjm8\" (UniqueName: \"kubernetes.io/projected/fbeb3764-7ae9-41d9-8a2a-b6f14affeb06-kube-api-access-6kjm8\") pod \"maas-api-revoke-sub-6b19338f-hp9zg\" (UID: \"fbeb3764-7ae9-41d9-8a2a-b6f14affeb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hp9zg"
Oct 30 22:44:56.705796 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:56.705658    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbeb3764-7ae9-41d9-8a2a-b6f14affeb06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-hp9zg\" (UID: \"fbeb3764-7ae9-41d9-8a2a-b6f14affeb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hp9zg"
Oct 30 22:44:56.712687 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:56.707059    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbeb3764-7ae9-41d9-8a2a-b6f14affeb06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-hp9zg\" (UID: \"fbeb3764-7ae9-41d9-8a2a-b6f14affeb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hp9zg"
Oct 30 22:44:56.715203 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:56.715179    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbeb3764-7ae9-41d9-8a2a-b6f14affeb06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-hp9zg\" (UID: \"fbeb3764-7ae9-41d9-8a2a-b6f14affeb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hp9zg"
Oct 30 22:44:56.763292 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:56.763257    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kjm8\" (UniqueName: \"kubernetes.io/projected/fbeb3764-7ae9-41d9-8a2a-b6f14affeb06-kube-api-access-6kjm8\") pod \"maas-api-revoke-sub-6b19338f-hp9zg\" (UID: \"fbeb3764-7ae9-41d9-8a2a-b6f14affeb06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hp9zg"
Oct 30 22:44:56.904831 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:56.904751    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hp9zg"
Oct 30 22:44:57.279331 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:57.279252    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hp9zg"]
Oct 30 22:44:57.282237 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:44:57.282202    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbeb3764_7ae9_41d9_8a2a_b6f14affeb06.slice/crio-72b907317844cbb0eb40b1204ee79c1da09167dc067cda974ed3683104e7c03b WatchSource:0}: Error finding container 72b907317844cbb0eb40b1204ee79c1da09167dc067cda974ed3683104e7c03b: Status 404 returned error can't find the container with id 72b907317844cbb0eb40b1204ee79c1da09167dc067cda974ed3683104e7c03b
Oct 30 22:44:57.838528 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:57.838433    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hp9zg" event={"ID":"fbeb3764-7ae9-41d9-8a2a-b6f14affeb06","Type":"ContainerStarted","Data":"ea55e13ec73f199c3721b514ba1d1f5a51f9ed8e66f62ced72c2f8876c247516"}
Oct 30 22:44:57.838528 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:57.838491    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hp9zg" event={"ID":"fbeb3764-7ae9-41d9-8a2a-b6f14affeb06","Type":"ContainerStarted","Data":"72b907317844cbb0eb40b1204ee79c1da09167dc067cda974ed3683104e7c03b"}
Oct 30 22:44:57.963009 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:57.962958    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hp9zg" podStartSLOduration=1.962945109 podStartE2EDuration="1.962945109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:57.960806471 +0000 UTC m=+4292.435310450" watchObservedRunningTime="2026-10-30 22:44:57.962945109 +0000 UTC m=+4292.437449088"
Oct 30 22:44:58.844446 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:58.844407    5087 generic.go:356] "Generic (PLEG): container finished" podID="fbeb3764-7ae9-41d9-8a2a-b6f14affeb06" containerID="ea55e13ec73f199c3721b514ba1d1f5a51f9ed8e66f62ced72c2f8876c247516" exitCode=0
Oct 30 22:44:58.844854 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:58.844497    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hp9zg" event={"ID":"fbeb3764-7ae9-41d9-8a2a-b6f14affeb06","Type":"ContainerDied","Data":"ea55e13ec73f199c3721b514ba1d1f5a51f9ed8e66f62ced72c2f8876c247516"}
Oct 30 22:44:59.991956 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:44:59.991932    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hp9zg"
Oct 30 22:45:00.039966 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:00.039938    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbeb3764-7ae9-41d9-8a2a-b6f14affeb06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbeb3764-7ae9-41d9-8a2a-b6f14affeb06-maas-api-openshift-ca\") pod \"fbeb3764-7ae9-41d9-8a2a-b6f14affeb06\" (UID: \"fbeb3764-7ae9-41d9-8a2a-b6f14affeb06\") "
Oct 30 22:45:00.040103 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:00.040013    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbeb3764-7ae9-41d9-8a2a-b6f14affeb06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbeb3764-7ae9-41d9-8a2a-b6f14affeb06-maas-api-xks-ca\") pod \"fbeb3764-7ae9-41d9-8a2a-b6f14affeb06\" (UID: \"fbeb3764-7ae9-41d9-8a2a-b6f14affeb06\") "
Oct 30 22:45:00.040103 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:00.040033    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbeb3764-7ae9-41d9-8a2a-b6f14affeb06-kube-api-access-6kjm8\" (UniqueName: \"kubernetes.io/projected/fbeb3764-7ae9-41d9-8a2a-b6f14affeb06-kube-api-access-6kjm8\") pod \"fbeb3764-7ae9-41d9-8a2a-b6f14affeb06\" (UID: \"fbeb3764-7ae9-41d9-8a2a-b6f14affeb06\") "
Oct 30 22:45:00.040286 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:00.040264    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbeb3764-7ae9-41d9-8a2a-b6f14affeb06-maas-api-openshift-ca" pod "fbeb3764-7ae9-41d9-8a2a-b6f14affeb06" (UID: "fbeb3764-7ae9-41d9-8a2a-b6f14affeb06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:00.042422 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:00.042313    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbeb3764-7ae9-41d9-8a2a-b6f14affeb06-kube-api-access-6kjm8" pod "fbeb3764-7ae9-41d9-8a2a-b6f14affeb06" (UID: "fbeb3764-7ae9-41d9-8a2a-b6f14affeb06"). InnerVolumeSpecName "kube-api-access-6kjm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:00.042422 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:00.042322    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbeb3764-7ae9-41d9-8a2a-b6f14affeb06-maas-api-xks-ca" pod "fbeb3764-7ae9-41d9-8a2a-b6f14affeb06" (UID: "fbeb3764-7ae9-41d9-8a2a-b6f14affeb06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:45:00.140873 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:00.140796    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbeb3764-7ae9-41d9-8a2a-b6f14affeb06-maas-api-openshift-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:00.140873 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:00.140824    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbeb3764-7ae9-41d9-8a2a-b6f14affeb06-maas-api-xks-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:00.140873 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:00.140834    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kjm8\" (UniqueName: \"kubernetes.io/projected/fbeb3764-7ae9-41d9-8a2a-b6f14affeb06-kube-api-access-6kjm8\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:00.178714 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:00.178673    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-dzmj7"]
Oct 30 22:45:00.179195 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:00.179174    5087 cpu_manager.go:414] "containerMap: removing container" podUID="fbeb3764-7ae9-41d9-8a2a-b6f14affeb06" containerName="revoke-keys"
Oct 30 22:45:00.179289 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:00.179197    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbeb3764-7ae9-41d9-8a2a-b6f14affeb06" containerName="revoke-keys"
Oct 30 22:45:00.179373 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:00.179309    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbeb3764-7ae9-41d9-8a2a-b6f14affeb06" containerName="revoke-keys"
Oct 30 22:45:00.182554 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:00.182532    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-dzmj7"
Oct 30 22:45:00.240298 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:00.240263    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-dzmj7"]
Oct 30 22:45:00.242193 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:00.242157    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5kz2\" (UniqueName: \"kubernetes.io/projected/e3c4b56d-c366-4eed-98b9-67701fbf89d7-kube-api-access-c5kz2\") pod \"maas-api-key-cleanup-29890005-dzmj7\" (UID: \"e3c4b56d-c366-4eed-98b9-67701fbf89d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-dzmj7"
Oct 30 22:45:00.242401 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:00.242264    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3c4b56d-c366-4eed-98b9-67701fbf89d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-dzmj7\" (UID: \"e3c4b56d-c366-4eed-98b9-67701fbf89d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-dzmj7"
Oct 30 22:45:00.343182 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:00.343145    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5kz2\" (UniqueName: \"kubernetes.io/projected/e3c4b56d-c366-4eed-98b9-67701fbf89d7-kube-api-access-c5kz2\") pod \"maas-api-key-cleanup-29890005-dzmj7\" (UID: \"e3c4b56d-c366-4eed-98b9-67701fbf89d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-dzmj7"
Oct 30 22:45:00.343395 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:00.343223    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3c4b56d-c366-4eed-98b9-67701fbf89d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-dzmj7\" (UID: \"e3c4b56d-c366-4eed-98b9-67701fbf89d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-dzmj7"
Oct 30 22:45:00.343868 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:00.343848    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3c4b56d-c366-4eed-98b9-67701fbf89d7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-dzmj7\" (UID: \"e3c4b56d-c366-4eed-98b9-67701fbf89d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-dzmj7"
Oct 30 22:45:00.400749 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:00.400670    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5kz2\" (UniqueName: \"kubernetes.io/projected/e3c4b56d-c366-4eed-98b9-67701fbf89d7-kube-api-access-c5kz2\") pod \"maas-api-key-cleanup-29890005-dzmj7\" (UID: \"e3c4b56d-c366-4eed-98b9-67701fbf89d7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-dzmj7"
Oct 30 22:45:00.495715 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:00.495678    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-dzmj7"
Oct 30 22:45:00.854898 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:00.854867    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hp9zg"
Oct 30 22:45:00.855077 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:00.854899    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hp9zg" event={"ID":"fbeb3764-7ae9-41d9-8a2a-b6f14affeb06","Type":"ContainerDied","Data":"72b907317844cbb0eb40b1204ee79c1da09167dc067cda974ed3683104e7c03b"}
Oct 30 22:45:00.855077 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:00.854949    5087 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72b907317844cbb0eb40b1204ee79c1da09167dc067cda974ed3683104e7c03b"
Oct 30 22:45:00.868010 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:00.867827    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-dzmj7"]
Oct 30 22:45:00.870644 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:45:00.870620    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3c4b56d_c366_4eed_98b9_67701fbf89d7.slice/crio-cbdb44e1830de82a16c09f14efb17259cb1716b0e390e9bf2f3233729378bafd WatchSource:0}: Error finding container cbdb44e1830de82a16c09f14efb17259cb1716b0e390e9bf2f3233729378bafd: Status 404 returned error can't find the container with id cbdb44e1830de82a16c09f14efb17259cb1716b0e390e9bf2f3233729378bafd
Oct 30 22:45:01.861360 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:01.861319    5087 generic.go:356] "Generic (PLEG): container finished" podID="e3c4b56d-c366-4eed-98b9-67701fbf89d7" containerID="e04e17f01268cb98761185c3e31503e46da9f8aee79a951c4101fa737e84a156" exitCode=60
Oct 30 22:45:01.861799 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:01.861417    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-dzmj7" event={"ID":"e3c4b56d-c366-4eed-98b9-67701fbf89d7","Type":"ContainerDied","Data":"e04e17f01268cb98761185c3e31503e46da9f8aee79a951c4101fa737e84a156"}
Oct 30 22:45:01.861799 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:01.861479    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-dzmj7" event={"ID":"e3c4b56d-c366-4eed-98b9-67701fbf89d7","Type":"ContainerStarted","Data":"cbdb44e1830de82a16c09f14efb17259cb1716b0e390e9bf2f3233729378bafd"}
Oct 30 22:45:01.861799 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:01.861631    5087 scope.go:122] "RemoveContainer" containerID="e04e17f01268cb98761185c3e31503e46da9f8aee79a951c4101fa737e84a156"
Oct 30 22:45:02.868411 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:02.868379    5087 generic.go:356] "Generic (PLEG): container finished" podID="e3c4b56d-c366-4eed-98b9-67701fbf89d7" containerID="97435e2973578060141f339578a700cd9942fac315c2078c6e21fddb58466913" exitCode=60
Oct 30 22:45:02.868876 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:02.868427    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-dzmj7" event={"ID":"e3c4b56d-c366-4eed-98b9-67701fbf89d7","Type":"ContainerDied","Data":"97435e2973578060141f339578a700cd9942fac315c2078c6e21fddb58466913"}
Oct 30 22:45:02.868876 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:02.868472    5087 scope.go:122] "RemoveContainer" containerID="e04e17f01268cb98761185c3e31503e46da9f8aee79a951c4101fa737e84a156"
Oct 30 22:45:02.868876 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:02.868727    5087 scope.go:122] "RemoveContainer" containerID="97435e2973578060141f339578a700cd9942fac315c2078c6e21fddb58466913"
Oct 30 22:45:02.869011 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:45:02.868935    5087 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-dzmj7_odh-ai-gateway-infra(e3c4b56d-c366-4eed-98b9-67701fbf89d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-dzmj7" podUID="e3c4b56d-c366-4eed-98b9-67701fbf89d7"
Oct 30 22:45:03.874993 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:03.874964    5087 scope.go:122] "RemoveContainer" containerID="97435e2973578060141f339578a700cd9942fac315c2078c6e21fddb58466913"
Oct 30 22:45:03.875426 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:45:03.875184    5087 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-dzmj7_odh-ai-gateway-infra(e3c4b56d-c366-4eed-98b9-67701fbf89d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-dzmj7" podUID="e3c4b56d-c366-4eed-98b9-67701fbf89d7"
Oct 30 22:45:13.701811 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:13.701775    5087 scope.go:122] "RemoveContainer" containerID="97435e2973578060141f339578a700cd9942fac315c2078c6e21fddb58466913"
Oct 30 22:45:13.929514 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:13.929474    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-dzmj7" event={"ID":"e3c4b56d-c366-4eed-98b9-67701fbf89d7","Type":"ContainerStarted","Data":"fe37900068222c062c15c3aa0eb0e431ea93f639686ee5d1f5e83cc93f3cf873"}
Oct 30 22:45:14.056795 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:14.056710    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-dzmj7" podStartSLOduration=14.056695105 podStartE2EDuration="14.056695105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:14.056389366 +0000 UTC m=+4308.530893338" watchObservedRunningTime="2026-10-30 22:45:14.056695105 +0000 UTC m=+4308.531199084"
Oct 30 22:45:14.935749 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:14.935714    5087 generic.go:356] "Generic (PLEG): container finished" podID="e3c4b56d-c366-4eed-98b9-67701fbf89d7" containerID="fe37900068222c062c15c3aa0eb0e431ea93f639686ee5d1f5e83cc93f3cf873" exitCode=60
Oct 30 22:45:14.936227 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:14.935771    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-dzmj7" event={"ID":"e3c4b56d-c366-4eed-98b9-67701fbf89d7","Type":"ContainerDied","Data":"fe37900068222c062c15c3aa0eb0e431ea93f639686ee5d1f5e83cc93f3cf873"}
Oct 30 22:45:14.936227 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:14.935829    5087 scope.go:122] "RemoveContainer" containerID="97435e2973578060141f339578a700cd9942fac315c2078c6e21fddb58466913"
Oct 30 22:45:14.936227 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:14.936172    5087 scope.go:122] "RemoveContainer" containerID="fe37900068222c062c15c3aa0eb0e431ea93f639686ee5d1f5e83cc93f3cf873"
Oct 30 22:45:14.936460 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:45:14.936439    5087 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-dzmj7_odh-ai-gateway-infra(e3c4b56d-c366-4eed-98b9-67701fbf89d7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-dzmj7" podUID="e3c4b56d-c366-4eed-98b9-67701fbf89d7"
Oct 30 22:45:15.067624 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:15.067591    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-dzmj7"]
Oct 30 22:45:16.085986 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:16.085962    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-dzmj7"
Oct 30 22:45:16.229494 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:16.229422    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c4b56d-c366-4eed-98b9-67701fbf89d7-kube-api-access-c5kz2\" (UniqueName: \"kubernetes.io/projected/e3c4b56d-c366-4eed-98b9-67701fbf89d7-kube-api-access-c5kz2\") pod \"e3c4b56d-c366-4eed-98b9-67701fbf89d7\" (UID: \"e3c4b56d-c366-4eed-98b9-67701fbf89d7\") "
Oct 30 22:45:16.229494 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:16.229451    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3c4b56d-c366-4eed-98b9-67701fbf89d7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3c4b56d-c366-4eed-98b9-67701fbf89d7-maas-api-service-ca\") pod \"e3c4b56d-c366-4eed-98b9-67701fbf89d7\" (UID: \"e3c4b56d-c366-4eed-98b9-67701fbf89d7\") "
Oct 30 22:45:16.229874 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:16.229852    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3c4b56d-c366-4eed-98b9-67701fbf89d7-maas-api-service-ca" pod "e3c4b56d-c366-4eed-98b9-67701fbf89d7" (UID: "e3c4b56d-c366-4eed-98b9-67701fbf89d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:16.231798 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:16.231774    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c4b56d-c366-4eed-98b9-67701fbf89d7-kube-api-access-c5kz2" pod "e3c4b56d-c366-4eed-98b9-67701fbf89d7" (UID: "e3c4b56d-c366-4eed-98b9-67701fbf89d7"). InnerVolumeSpecName "kube-api-access-c5kz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:16.331158 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:16.331126    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5kz2\" (UniqueName: \"kubernetes.io/projected/e3c4b56d-c366-4eed-98b9-67701fbf89d7-kube-api-access-c5kz2\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:16.331158 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:16.331155    5087 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3c4b56d-c366-4eed-98b9-67701fbf89d7-maas-api-service-ca\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:45:16.949185 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:16.949155    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-dzmj7"
Oct 30 22:45:16.949385 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:16.949204    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-dzmj7" event={"ID":"e3c4b56d-c366-4eed-98b9-67701fbf89d7","Type":"ContainerDied","Data":"cbdb44e1830de82a16c09f14efb17259cb1716b0e390e9bf2f3233729378bafd"}
Oct 30 22:45:16.949385 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:16.949267    5087 scope.go:122] "RemoveContainer" containerID="fe37900068222c062c15c3aa0eb0e431ea93f639686ee5d1f5e83cc93f3cf873"
Oct 30 22:45:17.114046 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:17.114008    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-dzmj7"]
Oct 30 22:45:17.163053 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:17.163024    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-dzmj7"]
Oct 30 22:45:17.166304 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:45:17.166271    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:45:17.707978 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:17.707946    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3c4b56d-c366-4eed-98b9-67701fbf89d7" path="/var/lib/kubelet/pods/e3c4b56d-c366-4eed-98b9-67701fbf89d7/volumes"
Oct 30 22:45:30.110037 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:30.110003    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-mnddg_4319fba2-04c7-4072-806e-04db5ba1eeca/manager/0.log"
Oct 30 22:45:30.394676 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:30.394584    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-swnjl_03fd0d5a-b3d4-40c3-932c-af7a51959c3b/manager/0.log"
Oct 30 22:45:30.771530 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:30.771498    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-s24zk_b85cc940-d5e6-4869-9763-881acf0ea4ac/manager/0.log"
Oct 30 22:45:33.774134 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:33.774101    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-v4dg8_26516f1e-739c-427b-98cd-4e387fd462bb/revoke-keys/0.log"
Oct 30 22:45:33.996424 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:33.996392    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-hp9zg_fbeb3764-7ae9-41d9-8a2a-b6f14affeb06/revoke-keys/0.log"
Oct 30 22:45:34.223715 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:34.223636    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-266w9_07407208-6eca-43c8-936b-8cbcb88283cf/revoke-keys/0.log"
Oct 30 22:45:34.443587 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:34.443554    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-jvnl5_4117714f-de1f-4a99-9338-9837664d51ba/postgres/0.log"
Oct 30 22:45:36.473275 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:36.473242    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-mplpk_f6eeaec5-a9fd-48cc-bfb2-10e0699d4c2f/manager/0.log"
Oct 30 22:45:47.166381 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:45:47.166326    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:45:47.945672 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:45:47.945641    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-jdkxk_c599a977-d2f9-4589-b4df-8590b31744d3/manager/0.log"
Oct 30 22:46:08.921925 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:46:08.921845    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf_b05d7b91-0d6e-460e-b5c3-40a261c91e1f/istio-proxy/0.log"
Oct 30 22:46:11.361207 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:46:11.361170    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-74854d8ccf-pwfjh_9c6d8084-9482-438d-b5e5-3f36442423d2/istio-proxy/0.log"
Oct 30 22:46:11.870622 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:46:11.870590    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-74854d8ccf-v6jrr_41ec8a58-81a8-4586-8196-bb59d4bee8e4/istio-proxy/0.log"
Oct 30 22:46:12.395715 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:46:12.395672    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-74854d8ccf-zbxls_84240bba-5ff2-4f87-8f9e-d5350089d27e/istio-proxy/0.log"
Oct 30 22:46:12.631642 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:46:12.631612    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s_15eaf1f9-3376-4b99-b708-84a2d6af5d1b/istio-proxy/0.log"
Oct 30 22:46:13.298149 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:46:13.298117    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-54cbfc65d5-sg48h_83917e8e-50b6-4c52-b337-6f1be5a2c26a/router/0.log"
Oct 30 22:46:17.167156 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:46:17.167114    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:46:36.865881 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:46:36.865844    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-mnddg_4319fba2-04c7-4072-806e-04db5ba1eeca/manager/0.log"
Oct 30 22:46:37.150482 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:46:37.150394    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-swnjl_03fd0d5a-b3d4-40c3-932c-af7a51959c3b/manager/0.log"
Oct 30 22:46:37.494139 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:46:37.494105    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-s24zk_b85cc940-d5e6-4869-9763-881acf0ea4ac/manager/0.log"
Oct 30 22:46:39.655127 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:46:39.655100    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-v4dg8_26516f1e-739c-427b-98cd-4e387fd462bb/revoke-keys/0.log"
Oct 30 22:46:39.878035 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:46:39.878009    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-hp9zg_fbeb3764-7ae9-41d9-8a2a-b6f14affeb06/revoke-keys/0.log"
Oct 30 22:46:40.107812 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:46:40.107781    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-266w9_07407208-6eca-43c8-936b-8cbcb88283cf/revoke-keys/0.log"
Oct 30 22:46:40.331533 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:46:40.331502    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-jvnl5_4117714f-de1f-4a99-9338-9837664d51ba/postgres/0.log"
Oct 30 22:46:41.956606 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:46:41.956573    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-mplpk_f6eeaec5-a9fd-48cc-bfb2-10e0699d4c2f/manager/0.log"
Oct 30 22:46:43.244747 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:46:43.244713    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-jdkxk_c599a977-d2f9-4589-b4df-8590b31744d3/manager/0.log"
Oct 30 22:46:43.992458 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:46:43.992427    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf_b05d7b91-0d6e-460e-b5c3-40a261c91e1f/istio-proxy/0.log"
Oct 30 22:46:46.369392 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:46:46.369342    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-74854d8ccf-pwfjh_9c6d8084-9482-438d-b5e5-3f36442423d2/istio-proxy/0.log"
Oct 30 22:46:46.882020 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:46:46.881988    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-74854d8ccf-v6jrr_41ec8a58-81a8-4586-8196-bb59d4bee8e4/istio-proxy/0.log"
Oct 30 22:46:47.167766 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:46:47.167687    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:46:47.411258 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:46:47.411231    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-74854d8ccf-zbxls_84240bba-5ff2-4f87-8f9e-d5350089d27e/istio-proxy/0.log"
Oct 30 22:46:47.642996 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:46:47.642962    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-fc457bc99-gjm4s_15eaf1f9-3376-4b99-b708-84a2d6af5d1b/istio-proxy/0.log"
Oct 30 22:46:48.297194 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:46:48.297164    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-54cbfc65d5-sg48h_83917e8e-50b6-4c52-b337-6f1be5a2c26a/router/0.log"
Oct 30 22:47:17.167070 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:47:17.167019    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:47:30.477419 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:47:30.477385    5087 scope.go:122] "RemoveContainer" containerID="a4d45095e37d3aad5d049824164d89a682b9cab34a1c13f8e6e673669fcecf29"
Oct 30 22:47:47.167517 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:47:47.167435    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:48:17.167549 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:48:17.167440    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:48:26.383732 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:48:26.383694    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 22:48:26.388847 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:48:26.388824    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-z8clf_1196dab1-80d7-48eb-8d05-1a26ab6d0759/console-operator/1.log"
Oct 30 22:48:26.393057 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:48:26.393036    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 22:48:26.397650 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:48:26.397634    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-bjt99_7986120b-482f-4b0b-a605-32770569962d/ovn-acl-logging/0.log"
Oct 30 22:48:47.167179 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:48:47.167133    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:49:17.166785 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:49:17.166701    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:49:27.055888 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:49:27.055846    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-266w9"]
Oct 30 22:49:27.100672 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:49:27.100634    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-266w9"]
Oct 30 22:49:27.707834 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:49:27.707800    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07407208-6eca-43c8-936b-8cbcb88283cf" path="/var/lib/kubelet/pods/07407208-6eca-43c8-936b-8cbcb88283cf/volumes"
Oct 30 22:49:30.504203 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:49:30.504170    5087 scope.go:122] "RemoveContainer" containerID="bdb15dfbb86436bdc6cca8253a60b96932431fa755e95bedb4cbe6e53fa89e29"
Oct 30 22:49:47.166932 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:49:47.166897    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:49:49.058935 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:49:49.058899    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4dg8"]
Oct 30 22:49:49.108158 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:49:49.108126    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-v4dg8"]
Oct 30 22:49:49.708401 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:49:49.708369    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26516f1e-739c-427b-98cd-4e387fd462bb" path="/var/lib/kubelet/pods/26516f1e-739c-427b-98cd-4e387fd462bb/volumes"
Oct 30 22:50:01.047726 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:01.047690    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hp9zg"]
Oct 30 22:50:01.096930 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:01.096901    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-hp9zg"]
Oct 30 22:50:01.710641 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:01.710607    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbeb3764-7ae9-41d9-8a2a-b6f14affeb06" path="/var/lib/kubelet/pods/fbeb3764-7ae9-41d9-8a2a-b6f14affeb06/volumes"
Oct 30 22:50:17.167332 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:50:17.167288    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:50:20.247472 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:20.247442    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-phvzg/must-gather-2f4hw"]
Oct 30 22:50:20.247895 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:20.247875    5087 cpu_manager.go:414] "containerMap: removing container" podUID="e3c4b56d-c366-4eed-98b9-67701fbf89d7" containerName="cleanup"
Oct 30 22:50:20.247895 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:20.247886    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3c4b56d-c366-4eed-98b9-67701fbf89d7" containerName="cleanup"
Oct 30 22:50:20.247981 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:20.247913    5087 cpu_manager.go:414] "containerMap: removing container" podUID="e3c4b56d-c366-4eed-98b9-67701fbf89d7" containerName="cleanup"
Oct 30 22:50:20.247981 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:20.247918    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3c4b56d-c366-4eed-98b9-67701fbf89d7" containerName="cleanup"
Oct 30 22:50:20.248049 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:20.247986    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3c4b56d-c366-4eed-98b9-67701fbf89d7" containerName="cleanup"
Oct 30 22:50:20.248049 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:20.247996    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3c4b56d-c366-4eed-98b9-67701fbf89d7" containerName="cleanup"
Oct 30 22:50:20.248116 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:20.248075    5087 cpu_manager.go:414] "containerMap: removing container" podUID="e3c4b56d-c366-4eed-98b9-67701fbf89d7" containerName="cleanup"
Oct 30 22:50:20.248116 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:20.248081    5087 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3c4b56d-c366-4eed-98b9-67701fbf89d7" containerName="cleanup"
Oct 30 22:50:20.248180 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:20.248169    5087 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3c4b56d-c366-4eed-98b9-67701fbf89d7" containerName="cleanup"
Oct 30 22:50:20.251329 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:20.251309    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-phvzg/must-gather-2f4hw"
Oct 30 22:50:20.322160 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:20.322110    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/018f8b98-e52d-4048-bd33-be7567239337-must-gather-output\") pod \"must-gather-2f4hw\" (UID: \"018f8b98-e52d-4048-bd33-be7567239337\") " pod="openshift-must-gather-phvzg/must-gather-2f4hw"
Oct 30 22:50:20.322468 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:20.322445    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgxn2\" (UniqueName: \"kubernetes.io/projected/018f8b98-e52d-4048-bd33-be7567239337-kube-api-access-qgxn2\") pod \"must-gather-2f4hw\" (UID: \"018f8b98-e52d-4048-bd33-be7567239337\") " pod="openshift-must-gather-phvzg/must-gather-2f4hw"
Oct 30 22:50:20.352753 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:20.352724    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-phvzg\"/\"kube-root-ca.crt\""
Oct 30 22:50:20.352864 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:20.352784    5087 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-phvzg\"/\"openshift-service-ca.crt\""
Oct 30 22:50:20.352864 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:20.352827    5087 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-phvzg\"/\"default-dockercfg-t97cc\""
Oct 30 22:50:20.355042 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:20.355020    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-phvzg/must-gather-2f4hw"]
Oct 30 22:50:20.423704 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:20.423681    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgxn2\" (UniqueName: \"kubernetes.io/projected/018f8b98-e52d-4048-bd33-be7567239337-kube-api-access-qgxn2\") pod \"must-gather-2f4hw\" (UID: \"018f8b98-e52d-4048-bd33-be7567239337\") " pod="openshift-must-gather-phvzg/must-gather-2f4hw"
Oct 30 22:50:20.423841 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:20.423742    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/018f8b98-e52d-4048-bd33-be7567239337-must-gather-output\") pod \"must-gather-2f4hw\" (UID: \"018f8b98-e52d-4048-bd33-be7567239337\") " pod="openshift-must-gather-phvzg/must-gather-2f4hw"
Oct 30 22:50:20.424036 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:20.424020    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/018f8b98-e52d-4048-bd33-be7567239337-must-gather-output\") pod \"must-gather-2f4hw\" (UID: \"018f8b98-e52d-4048-bd33-be7567239337\") " pod="openshift-must-gather-phvzg/must-gather-2f4hw"
Oct 30 22:50:20.480521 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:20.480493    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgxn2\" (UniqueName: \"kubernetes.io/projected/018f8b98-e52d-4048-bd33-be7567239337-kube-api-access-qgxn2\") pod \"must-gather-2f4hw\" (UID: \"018f8b98-e52d-4048-bd33-be7567239337\") " pod="openshift-must-gather-phvzg/must-gather-2f4hw"
Oct 30 22:50:20.576209 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:20.576134    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-phvzg/must-gather-2f4hw"
Oct 30 22:50:20.946601 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:20.946515    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-phvzg/must-gather-2f4hw"]
Oct 30 22:50:20.948893 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:50:20.948870    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod018f8b98_e52d_4048_bd33_be7567239337.slice/crio-a324c287cf7b750ebef49e7a28049a79a5bdb18688ee089299db0aa6c6184a64 WatchSource:0}: Error finding container a324c287cf7b750ebef49e7a28049a79a5bdb18688ee089299db0aa6c6184a64: Status 404 returned error can't find the container with id a324c287cf7b750ebef49e7a28049a79a5bdb18688ee089299db0aa6c6184a64
Oct 30 22:50:20.950612 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:20.950598    5087 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:50:21.399229 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:21.399185    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-phvzg/must-gather-2f4hw" event={"ID":"018f8b98-e52d-4048-bd33-be7567239337","Type":"ContainerStarted","Data":"a324c287cf7b750ebef49e7a28049a79a5bdb18688ee089299db0aa6c6184a64"}
Oct 30 22:50:22.408066 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:22.408034    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-phvzg/must-gather-2f4hw" event={"ID":"018f8b98-e52d-4048-bd33-be7567239337","Type":"ContainerStarted","Data":"08d1a484c547e23536c4f69ad84e529acf5cba79f5b084b4e20f9d855befaa9c"}
Oct 30 22:50:23.415737 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:23.415704    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-phvzg/must-gather-2f4hw" event={"ID":"018f8b98-e52d-4048-bd33-be7567239337","Type":"ContainerStarted","Data":"152145e188688f7ef886de0355bcdec96e2bc35d3d5a5f92d8d90afc9b548c1f"}
Oct 30 22:50:23.530766 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:23.530713    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-phvzg/must-gather-2f4hw" podStartSLOduration=2.285938533 podStartE2EDuration="3.530697048s" totalImagesPullingTime="1.244758515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:23.529426497 +0000 UTC m=+4618.003930470" watchObservedRunningTime="2026-10-30 22:50:23.530697048 +0000 UTC m=+4618.005201089"
Oct 30 22:50:29.744247 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:29.744218    5087 ???:1] "http: TLS handshake error from 10.0.132.92:41312: EOF"
Oct 30 22:50:29.753174 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:29.753140    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-2z8sh_2a95f3f4-cfa1-4ae6-b91b-bc20f1ccbb49/global-pull-secret-syncer/0.log"
Oct 30 22:50:30.527370 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:30.526465    5087 scope.go:122] "RemoveContainer" containerID="68d5d30aebb5ea3082e17506a512bacbf22bef8e02b4196c9d89ce5a0b893047"
Oct 30 22:50:30.989016 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:30.988388    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-9m2lz_8a74b249-57da-4405-af8e-ae777a44415e/connection-checker/0.log"
Oct 30 22:50:31.422029 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:31.421954    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-9trcs_364111d6-7289-47be-8cf0-c10c7b6fbcdc/connection-checker/0.log"
Oct 30 22:50:31.882459 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:31.882430    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-cx7d4_f84984eb-c144-4394-bb77-afc0c94b3a3a/connection-checker/0.log"
Oct 30 22:50:32.829375 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:32.827457    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-h5xkl_6e5648ef-56ca-4d99-ad1a-0adb0fc50855/konnectivity-agent/0.log"
Oct 30 22:50:34.206106 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:34.206075    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-92.us-west-1.compute.internal_57c3091500d45dc621d6409761c7c97d/haproxy/0.log"
Oct 30 22:50:38.830481 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:38.830445    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-mplpk_f6eeaec5-a9fd-48cc-bfb2-10e0699d4c2f/manager/0.log"
Oct 30 22:50:39.145978 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:39.145901    5087 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-phvzg/perf-node-gather-daemonset-kg52t"]
Oct 30 22:50:39.153191 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:39.153168    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-phvzg/perf-node-gather-daemonset-kg52t"
Oct 30 22:50:39.236606 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:39.236569    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-phvzg/perf-node-gather-daemonset-kg52t"]
Oct 30 22:50:39.316123 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:39.316087    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4a9568b-3d5f-46bc-949d-3946536a55f8-lib-modules\") pod \"perf-node-gather-daemonset-kg52t\" (UID: \"f4a9568b-3d5f-46bc-949d-3946536a55f8\") " pod="openshift-must-gather-phvzg/perf-node-gather-daemonset-kg52t"
Oct 30 22:50:39.316321 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:39.316167    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f4a9568b-3d5f-46bc-949d-3946536a55f8-proc\") pod \"perf-node-gather-daemonset-kg52t\" (UID: \"f4a9568b-3d5f-46bc-949d-3946536a55f8\") " pod="openshift-must-gather-phvzg/perf-node-gather-daemonset-kg52t"
Oct 30 22:50:39.316321 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:39.316284    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f4a9568b-3d5f-46bc-949d-3946536a55f8-podres\") pod \"perf-node-gather-daemonset-kg52t\" (UID: \"f4a9568b-3d5f-46bc-949d-3946536a55f8\") " pod="openshift-must-gather-phvzg/perf-node-gather-daemonset-kg52t"
Oct 30 22:50:39.316477 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:39.316404    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g62t4\" (UniqueName: \"kubernetes.io/projected/f4a9568b-3d5f-46bc-949d-3946536a55f8-kube-api-access-g62t4\") pod \"perf-node-gather-daemonset-kg52t\" (UID: \"f4a9568b-3d5f-46bc-949d-3946536a55f8\") " pod="openshift-must-gather-phvzg/perf-node-gather-daemonset-kg52t"
Oct 30 22:50:39.316477 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:39.316433    5087 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4a9568b-3d5f-46bc-949d-3946536a55f8-sys\") pod \"perf-node-gather-daemonset-kg52t\" (UID: \"f4a9568b-3d5f-46bc-949d-3946536a55f8\") " pod="openshift-must-gather-phvzg/perf-node-gather-daemonset-kg52t"
Oct 30 22:50:39.417571 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:39.417479    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g62t4\" (UniqueName: \"kubernetes.io/projected/f4a9568b-3d5f-46bc-949d-3946536a55f8-kube-api-access-g62t4\") pod \"perf-node-gather-daemonset-kg52t\" (UID: \"f4a9568b-3d5f-46bc-949d-3946536a55f8\") " pod="openshift-must-gather-phvzg/perf-node-gather-daemonset-kg52t"
Oct 30 22:50:39.417571 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:39.417525    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4a9568b-3d5f-46bc-949d-3946536a55f8-sys\") pod \"perf-node-gather-daemonset-kg52t\" (UID: \"f4a9568b-3d5f-46bc-949d-3946536a55f8\") " pod="openshift-must-gather-phvzg/perf-node-gather-daemonset-kg52t"
Oct 30 22:50:39.417571 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:39.417557    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4a9568b-3d5f-46bc-949d-3946536a55f8-lib-modules\") pod \"perf-node-gather-daemonset-kg52t\" (UID: \"f4a9568b-3d5f-46bc-949d-3946536a55f8\") " pod="openshift-must-gather-phvzg/perf-node-gather-daemonset-kg52t"
Oct 30 22:50:39.417873 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:39.417595    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f4a9568b-3d5f-46bc-949d-3946536a55f8-proc\") pod \"perf-node-gather-daemonset-kg52t\" (UID: \"f4a9568b-3d5f-46bc-949d-3946536a55f8\") " pod="openshift-must-gather-phvzg/perf-node-gather-daemonset-kg52t"
Oct 30 22:50:39.417873 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:39.417654    5087 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f4a9568b-3d5f-46bc-949d-3946536a55f8-podres\") pod \"perf-node-gather-daemonset-kg52t\" (UID: \"f4a9568b-3d5f-46bc-949d-3946536a55f8\") " pod="openshift-must-gather-phvzg/perf-node-gather-daemonset-kg52t"
Oct 30 22:50:39.417873 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:39.417677    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4a9568b-3d5f-46bc-949d-3946536a55f8-sys\") pod \"perf-node-gather-daemonset-kg52t\" (UID: \"f4a9568b-3d5f-46bc-949d-3946536a55f8\") " pod="openshift-must-gather-phvzg/perf-node-gather-daemonset-kg52t"
Oct 30 22:50:39.417873 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:39.417691    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4a9568b-3d5f-46bc-949d-3946536a55f8-lib-modules\") pod \"perf-node-gather-daemonset-kg52t\" (UID: \"f4a9568b-3d5f-46bc-949d-3946536a55f8\") " pod="openshift-must-gather-phvzg/perf-node-gather-daemonset-kg52t"
Oct 30 22:50:39.417873 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:39.417746    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f4a9568b-3d5f-46bc-949d-3946536a55f8-proc\") pod \"perf-node-gather-daemonset-kg52t\" (UID: \"f4a9568b-3d5f-46bc-949d-3946536a55f8\") " pod="openshift-must-gather-phvzg/perf-node-gather-daemonset-kg52t"
Oct 30 22:50:39.417873 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:39.417799    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f4a9568b-3d5f-46bc-949d-3946536a55f8-podres\") pod \"perf-node-gather-daemonset-kg52t\" (UID: \"f4a9568b-3d5f-46bc-949d-3946536a55f8\") " pod="openshift-must-gather-phvzg/perf-node-gather-daemonset-kg52t"
Oct 30 22:50:39.474872 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:39.474836    5087 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g62t4\" (UniqueName: \"kubernetes.io/projected/f4a9568b-3d5f-46bc-949d-3946536a55f8-kube-api-access-g62t4\") pod \"perf-node-gather-daemonset-kg52t\" (UID: \"f4a9568b-3d5f-46bc-949d-3946536a55f8\") " pod="openshift-must-gather-phvzg/perf-node-gather-daemonset-kg52t"
Oct 30 22:50:39.768946 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:39.768915    5087 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-phvzg/perf-node-gather-daemonset-kg52t"
Oct 30 22:50:40.143168 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:40.143061    5087 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-phvzg/perf-node-gather-daemonset-kg52t"]
Oct 30 22:50:40.145867 ip-10-0-132-92 kubenswrapper[5087]: W1030 22:50:40.145818    5087 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf4a9568b_3d5f_46bc_949d_3946536a55f8.slice/crio-2f18dca0733af81d75d7984c400f7b1c9a211487089d4532481b11d06b309e70 WatchSource:0}: Error finding container 2f18dca0733af81d75d7984c400f7b1c9a211487089d4532481b11d06b309e70: Status 404 returned error can't find the container with id 2f18dca0733af81d75d7984c400f7b1c9a211487089d4532481b11d06b309e70
Oct 30 22:50:40.531505 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:40.531467    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-phvzg/perf-node-gather-daemonset-kg52t" event={"ID":"f4a9568b-3d5f-46bc-949d-3946536a55f8","Type":"ContainerStarted","Data":"aae9552e7c0b12959799eb2a018bdd4adecc1682be2742985094f25f1d8e9307"}
Oct 30 22:50:40.531709 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:40.531521    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-phvzg/perf-node-gather-daemonset-kg52t" event={"ID":"f4a9568b-3d5f-46bc-949d-3946536a55f8","Type":"ContainerStarted","Data":"2f18dca0733af81d75d7984c400f7b1c9a211487089d4532481b11d06b309e70"}
Oct 30 22:50:40.531709 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:40.531624    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-phvzg/perf-node-gather-daemonset-kg52t"
Oct 30 22:50:40.645748 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:40.645686    5087 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-phvzg/perf-node-gather-daemonset-kg52t" podStartSLOduration=1.6456720790000001 podStartE2EDuration="1.645672079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:50:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:40.644680222 +0000 UTC m=+4635.119184202" watchObservedRunningTime="2026-10-30 22:50:40.645672079 +0000 UTC m=+4635.120176058"
Oct 30 22:50:40.944301 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:40.944234    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-jdkxk_c599a977-d2f9-4589-b4df-8590b31744d3/manager/0.log"
Oct 30 22:50:41.398065 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:41.398035    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"]
Oct 30 22:50:41.398483 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:41.398228    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls" podUID="84240bba-5ff2-4f87-8f9e-d5350089d27e" containerName="istio-proxy" containerID="cri-o://a381b1f224ca85ed41633366225d39686ee0c22f9b720ff970eaa2dd7e683c17" gracePeriod=30
Oct 30 22:50:41.449901 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:41.449808    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"]
Oct 30 22:50:41.450104 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:41.450026    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh" podUID="9c6d8084-9482-438d-b5e5-3f36442423d2" containerName="istio-proxy" containerID="cri-o://3ac6e46a230b2d2899daa347f3b1f02aaf3b31b6d9150fb29535dfb0b769fe5b" gracePeriod=30
Oct 30 22:50:41.451116 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:41.451090    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"]
Oct 30 22:50:41.451301 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:41.451280    5087 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr" podUID="41ec8a58-81a8-4586-8196-bb59d4bee8e4" containerName="istio-proxy" containerID="cri-o://7ae8da5bad16087af3ff4a4af5c54d654242d5a1f7e13c227080ceebf57d0594" gracePeriod=30
Oct 30 22:50:46.416199 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.416166    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-2qz48_9f16bd1f-a3bd-45ee-a02e-cb6b9e982327/cluster-monitoring-operator/0.log"
Oct 30 22:50:46.546740 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.546706    5087 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-phvzg/perf-node-gather-daemonset-kg52t"
Oct 30 22:50:46.570734 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.570699    5087 generic.go:356] "Generic (PLEG): container finished" podID="41ec8a58-81a8-4586-8196-bb59d4bee8e4" containerID="7ae8da5bad16087af3ff4a4af5c54d654242d5a1f7e13c227080ceebf57d0594" exitCode=0
Oct 30 22:50:46.570869 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.570835    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr" event={"ID":"41ec8a58-81a8-4586-8196-bb59d4bee8e4","Type":"ContainerDied","Data":"7ae8da5bad16087af3ff4a4af5c54d654242d5a1f7e13c227080ceebf57d0594"}
Oct 30 22:50:46.572728 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.572702    5087 generic.go:356] "Generic (PLEG): container finished" podID="84240bba-5ff2-4f87-8f9e-d5350089d27e" containerID="a381b1f224ca85ed41633366225d39686ee0c22f9b720ff970eaa2dd7e683c17" exitCode=0
Oct 30 22:50:46.572831 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.572758    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls" event={"ID":"84240bba-5ff2-4f87-8f9e-d5350089d27e","Type":"ContainerDied","Data":"a381b1f224ca85ed41633366225d39686ee0c22f9b720ff970eaa2dd7e683c17"}
Oct 30 22:50:46.574505 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.574483    5087 generic.go:356] "Generic (PLEG): container finished" podID="9c6d8084-9482-438d-b5e5-3f36442423d2" containerID="3ac6e46a230b2d2899daa347f3b1f02aaf3b31b6d9150fb29535dfb0b769fe5b" exitCode=0
Oct 30 22:50:46.574628 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.574573    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh" event={"ID":"9c6d8084-9482-438d-b5e5-3f36442423d2","Type":"ContainerDied","Data":"3ac6e46a230b2d2899daa347f3b1f02aaf3b31b6d9150fb29535dfb0b769fe5b"}
Oct 30 22:50:46.735404 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.735324    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:50:46.763394 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.760276    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:50:46.791890 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.791647    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-istio-data\") pod \"84240bba-5ff2-4f87-8f9e-d5350089d27e\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") "
Oct 30 22:50:46.791890 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.791702    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-workload-certs\") pod \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") "
Oct 30 22:50:46.791890 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.791738    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istio-envoy\") pod \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") "
Oct 30 22:50:46.791890 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.791758    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-credential-socket\") pod \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") "
Oct 30 22:50:46.791890 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.791790    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-credential-socket\") pod \"84240bba-5ff2-4f87-8f9e-d5350089d27e\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") "
Oct 30 22:50:46.791890 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.791821    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84240bba-5ff2-4f87-8f9e-d5350089d27e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84240bba-5ff2-4f87-8f9e-d5350089d27e-istiod-ca-cert\") pod \"84240bba-5ff2-4f87-8f9e-d5350089d27e\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") "
Oct 30 22:50:46.791890 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.791883    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-workload-socket\") pod \"84240bba-5ff2-4f87-8f9e-d5350089d27e\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") "
Oct 30 22:50:46.792403 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.791910    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84240bba-5ff2-4f87-8f9e-d5350089d27e-istio-token\" (UniqueName: \"kubernetes.io/projected/84240bba-5ff2-4f87-8f9e-d5350089d27e-istio-token\") pod \"84240bba-5ff2-4f87-8f9e-d5350089d27e\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") "
Oct 30 22:50:46.792403 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.791929    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84240bba-5ff2-4f87-8f9e-d5350089d27e-kube-api-access-4fpbd\" (UniqueName: \"kubernetes.io/projected/84240bba-5ff2-4f87-8f9e-d5350089d27e-kube-api-access-4fpbd\") pod \"84240bba-5ff2-4f87-8f9e-d5350089d27e\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") "
Oct 30 22:50:46.792403 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.791933    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-istio-data" pod "84240bba-5ff2-4f87-8f9e-d5350089d27e" (UID: "84240bba-5ff2-4f87-8f9e-d5350089d27e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:46.792403 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.791948    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istio-data\") pod \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") "
Oct 30 22:50:46.792403 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.792003    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-istio-envoy\") pod \"84240bba-5ff2-4f87-8f9e-d5350089d27e\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") "
Oct 30 22:50:46.792403 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.792058    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84240bba-5ff2-4f87-8f9e-d5350089d27e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84240bba-5ff2-4f87-8f9e-d5350089d27e-istio-podinfo\") pod \"84240bba-5ff2-4f87-8f9e-d5350089d27e\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") "
Oct 30 22:50:46.792403 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.792128    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-workload-certs\") pod \"84240bba-5ff2-4f87-8f9e-d5350089d27e\" (UID: \"84240bba-5ff2-4f87-8f9e-d5350089d27e\") "
Oct 30 22:50:46.792403 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.792145    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istio-data" pod "41ec8a58-81a8-4586-8196-bb59d4bee8e4" (UID: "41ec8a58-81a8-4586-8196-bb59d4bee8e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:46.792403 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.792154    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istio-podinfo\") pod \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") "
Oct 30 22:50:46.792403 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.792187    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-workload-socket\") pod \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") "
Oct 30 22:50:46.792403 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.792214    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istio-token\" (UniqueName: \"kubernetes.io/projected/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istio-token\") pod \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") "
Oct 30 22:50:46.792403 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.792256    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41ec8a58-81a8-4586-8196-bb59d4bee8e4-kube-api-access-q449h\" (UniqueName: \"kubernetes.io/projected/41ec8a58-81a8-4586-8196-bb59d4bee8e4-kube-api-access-q449h\") pod \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") "
Oct 30 22:50:46.792403 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.792282    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istiod-ca-cert\") pod \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\" (UID: \"41ec8a58-81a8-4586-8196-bb59d4bee8e4\") "
Oct 30 22:50:46.792403 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.792369    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-credential-socket" pod "41ec8a58-81a8-4586-8196-bb59d4bee8e4" (UID: "41ec8a58-81a8-4586-8196-bb59d4bee8e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:46.793127 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.792551    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-credential-socket" pod "84240bba-5ff2-4f87-8f9e-d5350089d27e" (UID: "84240bba-5ff2-4f87-8f9e-d5350089d27e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:46.793127 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.792726    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.793127 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.792746    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.793127 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.792761    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.793127 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.792774    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.793127 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.792780    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84240bba-5ff2-4f87-8f9e-d5350089d27e-istiod-ca-cert" pod "84240bba-5ff2-4f87-8f9e-d5350089d27e" (UID: "84240bba-5ff2-4f87-8f9e-d5350089d27e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:46.793127 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.792940    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-workload-socket" pod "84240bba-5ff2-4f87-8f9e-d5350089d27e" (UID: "84240bba-5ff2-4f87-8f9e-d5350089d27e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:46.793594 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.793569    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-workload-socket" pod "41ec8a58-81a8-4586-8196-bb59d4bee8e4" (UID: "41ec8a58-81a8-4586-8196-bb59d4bee8e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:46.794120 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.794096    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-workload-certs" pod "84240bba-5ff2-4f87-8f9e-d5350089d27e" (UID: "84240bba-5ff2-4f87-8f9e-d5350089d27e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:46.795016 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.794955    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-workload-certs" pod "41ec8a58-81a8-4586-8196-bb59d4bee8e4" (UID: "41ec8a58-81a8-4586-8196-bb59d4bee8e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:46.795178 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.795148    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istiod-ca-cert" pod "41ec8a58-81a8-4586-8196-bb59d4bee8e4" (UID: "41ec8a58-81a8-4586-8196-bb59d4bee8e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:46.795801 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.795777    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istio-envoy" pod "41ec8a58-81a8-4586-8196-bb59d4bee8e4" (UID: "41ec8a58-81a8-4586-8196-bb59d4bee8e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:46.796674 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.796647    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istio-podinfo" pod "41ec8a58-81a8-4586-8196-bb59d4bee8e4" (UID: "41ec8a58-81a8-4586-8196-bb59d4bee8e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:50:46.798467 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.798331    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istio-token" pod "41ec8a58-81a8-4586-8196-bb59d4bee8e4" (UID: "41ec8a58-81a8-4586-8196-bb59d4bee8e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:46.798612 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.798594    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84240bba-5ff2-4f87-8f9e-d5350089d27e-istio-podinfo" pod "84240bba-5ff2-4f87-8f9e-d5350089d27e" (UID: "84240bba-5ff2-4f87-8f9e-d5350089d27e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:50:46.798704 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.798685    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84240bba-5ff2-4f87-8f9e-d5350089d27e-istio-token" pod "84240bba-5ff2-4f87-8f9e-d5350089d27e" (UID: "84240bba-5ff2-4f87-8f9e-d5350089d27e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:46.798704 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.798686    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-istio-envoy" pod "84240bba-5ff2-4f87-8f9e-d5350089d27e" (UID: "84240bba-5ff2-4f87-8f9e-d5350089d27e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:46.798843 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.798714    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41ec8a58-81a8-4586-8196-bb59d4bee8e4-kube-api-access-q449h" pod "41ec8a58-81a8-4586-8196-bb59d4bee8e4" (UID: "41ec8a58-81a8-4586-8196-bb59d4bee8e4"). InnerVolumeSpecName "kube-api-access-q449h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:46.800685 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.800159    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84240bba-5ff2-4f87-8f9e-d5350089d27e-kube-api-access-4fpbd" pod "84240bba-5ff2-4f87-8f9e-d5350089d27e" (UID: "84240bba-5ff2-4f87-8f9e-d5350089d27e"). InnerVolumeSpecName "kube-api-access-4fpbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:46.803924 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.803905    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:50:46.893705 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.893679    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-workload-socket\") pod \"9c6d8084-9482-438d-b5e5-3f36442423d2\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") "
Oct 30 22:50:46.893905 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.893731    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c6d8084-9482-438d-b5e5-3f36442423d2-istio-token\" (UniqueName: \"kubernetes.io/projected/9c6d8084-9482-438d-b5e5-3f36442423d2-istio-token\") pod \"9c6d8084-9482-438d-b5e5-3f36442423d2\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") "
Oct 30 22:50:46.893905 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.893750    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c6d8084-9482-438d-b5e5-3f36442423d2-kube-api-access-hb5kb\" (UniqueName: \"kubernetes.io/projected/9c6d8084-9482-438d-b5e5-3f36442423d2-kube-api-access-hb5kb\") pod \"9c6d8084-9482-438d-b5e5-3f36442423d2\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") "
Oct 30 22:50:46.893905 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.893828    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-istio-envoy\") pod \"9c6d8084-9482-438d-b5e5-3f36442423d2\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") "
Oct 30 22:50:46.893905 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.893861    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c6d8084-9482-438d-b5e5-3f36442423d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6d8084-9482-438d-b5e5-3f36442423d2-istio-podinfo\") pod \"9c6d8084-9482-438d-b5e5-3f36442423d2\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") "
Oct 30 22:50:46.893905 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.893882    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-istio-data\") pod \"9c6d8084-9482-438d-b5e5-3f36442423d2\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") "
Oct 30 22:50:46.893905 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.893901    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-workload-certs\") pod \"9c6d8084-9482-438d-b5e5-3f36442423d2\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") "
Oct 30 22:50:46.894258 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.893920    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-credential-socket\") pod \"9c6d8084-9482-438d-b5e5-3f36442423d2\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") "
Oct 30 22:50:46.894258 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.893969    5087 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c6d8084-9482-438d-b5e5-3f36442423d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6d8084-9482-438d-b5e5-3f36442423d2-istiod-ca-cert\") pod \"9c6d8084-9482-438d-b5e5-3f36442423d2\" (UID: \"9c6d8084-9482-438d-b5e5-3f36442423d2\") "
Oct 30 22:50:46.894258 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.894016    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-workload-socket" pod "9c6d8084-9482-438d-b5e5-3f36442423d2" (UID: "9c6d8084-9482-438d-b5e5-3f36442423d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:46.894258 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.894234    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-workload-certs" pod "9c6d8084-9482-438d-b5e5-3f36442423d2" (UID: "9c6d8084-9482-438d-b5e5-3f36442423d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:46.894517 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.894297    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.894517 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.894316    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84240bba-5ff2-4f87-8f9e-d5350089d27e-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.894517 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.894319    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-credential-socket" pod "9c6d8084-9482-438d-b5e5-3f36442423d2" (UID: "9c6d8084-9482-438d-b5e5-3f36442423d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:46.894517 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.894334    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fpbd\" (UniqueName: \"kubernetes.io/projected/84240bba-5ff2-4f87-8f9e-d5350089d27e-kube-api-access-4fpbd\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.894517 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.894371    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.894517 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.894386    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84240bba-5ff2-4f87-8f9e-d5350089d27e-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.894517 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.894404    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84240bba-5ff2-4f87-8f9e-d5350089d27e-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.894517 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.894407    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-istio-data" pod "9c6d8084-9482-438d-b5e5-3f36442423d2" (UID: "9c6d8084-9482-438d-b5e5-3f36442423d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:46.894517 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.894418    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.894517 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.894433    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.894517 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.894447    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.894517 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.894461    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q449h\" (UniqueName: \"kubernetes.io/projected/41ec8a58-81a8-4586-8196-bb59d4bee8e4-kube-api-access-q449h\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.894517 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.894461    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c6d8084-9482-438d-b5e5-3f36442423d2-istiod-ca-cert" pod "9c6d8084-9482-438d-b5e5-3f36442423d2" (UID: "9c6d8084-9482-438d-b5e5-3f36442423d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:46.894517 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.894476    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.894517 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.894489    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.894517 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.894501    5087 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-workload-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.894517 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.894515    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41ec8a58-81a8-4586-8196-bb59d4bee8e4-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.894517 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.894528    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84240bba-5ff2-4f87-8f9e-d5350089d27e-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.896428 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.896329    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-istio-envoy" pod "9c6d8084-9482-438d-b5e5-3f36442423d2" (UID: "9c6d8084-9482-438d-b5e5-3f36442423d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:46.896536 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.896426    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c6d8084-9482-438d-b5e5-3f36442423d2-istio-token" pod "9c6d8084-9482-438d-b5e5-3f36442423d2" (UID: "9c6d8084-9482-438d-b5e5-3f36442423d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:46.896588 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.896559    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c6d8084-9482-438d-b5e5-3f36442423d2-istio-podinfo" pod "9c6d8084-9482-438d-b5e5-3f36442423d2" (UID: "9c6d8084-9482-438d-b5e5-3f36442423d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:50:46.896738 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.896713    5087 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c6d8084-9482-438d-b5e5-3f36442423d2-kube-api-access-hb5kb" pod "9c6d8084-9482-438d-b5e5-3f36442423d2" (UID: "9c6d8084-9482-438d-b5e5-3f36442423d2"). InnerVolumeSpecName "kube-api-access-hb5kb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:46.995923 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.995879    5087 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c6d8084-9482-438d-b5e5-3f36442423d2-istiod-ca-cert\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.995923 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.995922    5087 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c6d8084-9482-438d-b5e5-3f36442423d2-istio-token\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.996135 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.995941    5087 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb5kb\" (UniqueName: \"kubernetes.io/projected/9c6d8084-9482-438d-b5e5-3f36442423d2-kube-api-access-hb5kb\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.996135 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.995956    5087 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-istio-envoy\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.996135 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.995984    5087 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c6d8084-9482-438d-b5e5-3f36442423d2-istio-podinfo\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.996135 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.995999    5087 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-istio-data\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.996135 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.996015    5087 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-workload-certs\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:46.996135 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:46.996029    5087 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c6d8084-9482-438d-b5e5-3f36442423d2-credential-socket\") on node \"ip-10-0-132-92.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 22:50:47.171016 ip-10-0-132-92 kubenswrapper[5087]: E1030 22:50:47.170978    5087 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-g27zs" containerName="extractor"
Oct 30 22:50:47.596440 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:47.591640    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr" event={"ID":"41ec8a58-81a8-4586-8196-bb59d4bee8e4","Type":"ContainerDied","Data":"fc57968c78bb91ad65784ef2ed2d2c534693dd78bd8767950979fedf0f290c55"}
Oct 30 22:50:47.596440 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:47.591756    5087 scope.go:122] "RemoveContainer" containerID="7ae8da5bad16087af3ff4a4af5c54d654242d5a1f7e13c227080ceebf57d0594"
Oct 30 22:50:47.596440 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:47.591965    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"
Oct 30 22:50:47.597424 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:47.596801    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls" event={"ID":"84240bba-5ff2-4f87-8f9e-d5350089d27e","Type":"ContainerDied","Data":"ac62d5d52fcf77ffa91a72516809123c61767416ebd3093b05bd3a046a581265"}
Oct 30 22:50:47.599376 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:47.597475    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"
Oct 30 22:50:47.605368 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:47.604696    5087 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh" event={"ID":"9c6d8084-9482-438d-b5e5-3f36442423d2","Type":"ContainerDied","Data":"483415df9aaae7530af93e69decc3564fa76351faf07d425c841e66bba087ff1"}
Oct 30 22:50:47.605368 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:47.604809    5087 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"
Oct 30 22:50:47.608533 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:47.608510    5087 scope.go:122] "RemoveContainer" containerID="a381b1f224ca85ed41633366225d39686ee0c22f9b720ff970eaa2dd7e683c17"
Oct 30 22:50:47.625977 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:47.625947    5087 scope.go:122] "RemoveContainer" containerID="3ac6e46a230b2d2899daa347f3b1f02aaf3b31b6d9150fb29535dfb0b769fe5b"
Oct 30 22:50:47.766942 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:47.766904    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"]
Oct 30 22:50:47.816496 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:47.816460    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-zbxls"]
Oct 30 22:50:47.881232 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:47.881201    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"]
Oct 30 22:50:47.886571 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:47.886543    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-pwfjh"]
Oct 30 22:50:47.996220 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:47.996186    5087 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"]
Oct 30 22:50:47.999739 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:47.999715    5087 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-74854d8ccf-v6jrr"]
Oct 30 22:50:48.956187 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:48.956160    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bczf4_50d6c5b6-5d77-4098-8635-b4f38cae132f/node-exporter/0.log"
Oct 30 22:50:49.370438 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:49.370414    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bczf4_50d6c5b6-5d77-4098-8635-b4f38cae132f/kube-rbac-proxy/0.log"
Oct 30 22:50:49.707980 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:49.707917    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41ec8a58-81a8-4586-8196-bb59d4bee8e4" path="/var/lib/kubelet/pods/41ec8a58-81a8-4586-8196-bb59d4bee8e4/volumes"
Oct 30 22:50:49.708425 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:49.708407    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84240bba-5ff2-4f87-8f9e-d5350089d27e" path="/var/lib/kubelet/pods/84240bba-5ff2-4f87-8f9e-d5350089d27e/volumes"
Oct 30 22:50:49.708846 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:49.708833    5087 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c6d8084-9482-438d-b5e5-3f36442423d2" path="/var/lib/kubelet/pods/9c6d8084-9482-438d-b5e5-3f36442423d2/volumes"
Oct 30 22:50:49.782564 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:49.782545    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bczf4_50d6c5b6-5d77-4098-8635-b4f38cae132f/init-textfile/0.log"
Oct 30 22:50:57.724207 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:50:57.724177    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-hmplx_7589a4df-2be1-47b1-836f-09ba72813d82/prometheus-operator-admission-webhook/0.log"
Oct 30 22:51:01.863841 ip-10-0-132-92 kubenswrapper[5087]: I1030 22:51:01.863792    5087 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-576b8db9c7m9wmf_b05d7b91-0d6e-460e-b5c3-40a261c91e1f/istio-proxy/0.log"